Sasha Sirotkin

This is more of a cautionary tale than anything else. The lessons are obvious in hindsight: “use what you know” and “be careful of new technology”. What I want to do is explain how I fell into making these mistakes so others can avoid them.

Objectives

So, I wanted the infrastructure for my night-and-weekends project to be reasonably lean in terms of cost and complexity. For reference, I use a lot of AWS at my day job and it’s not cheap. I wanted to explore other cloud providers and had the following wish list:

1. It should have Terraform support. This enables disposable staging environments which is a great way to reduce costs. Terraform generally makes setting up infrastructure faster and safer as well. Plus, I have previous experience with Terraform.
2. It should support creating images through Packer. Each release is going to be image/AMI with production code baked into it. I use Docker at my day job, but it adds too much overhead due to orchestration and images are portable enough. I have previous experience with Packer as well.
3. It should offer a managed load balancer service that can be configured via Terraform. This makes it easy to add/remove instances without downtime which is useful since the releases are Packer-created images (this is essentially blue-green deployments).
4. It should offer a managed database service. If something terrible happens there should be a replica at the ready and at least a week’s worth of backups. I don’t want to worry about accidentally losing data.
5. It should be significantly cheaper. This eliminates the other major AWS-like cloud providers.

DigitalOcean caught my attention for a few reasons. First, it is half the cost of AWS. Droplets are cheap and provide remarkably good compute value. Second, they fulfill all the technical requirements above and are actually the only small cloud provider that offer managed databases as a service. Third, as an indie developer it feels more indie-like to use a smaller provider.

Stormy Waters

The documentation was straight forward and I was able to get a DigitalOcean fully environment running in a handful of evenings. Most of the time was actually spent setting up the image with Packer. It is not soon after that I started having problems.

I ended up having significant issues with DigitalOcean’s database and load balancer services. Droplets had 50% chance of not automatically being given network access to the database which meant I needed to do it manually each time. Worse yet, the load balancer started to give me more and more 503s as time went on as well. My guess was it was not properly removing droplets after they were being deleted (I was constantly creating and deleting droplets as part of my releases). Plus, there were other bugs too! I didn’t want to spend my precious time debugging all of this with their support.

As much as I want to blame everything on DigitalOcean, it was my fault as well. I am skeptical of using a brand new library in my code and I did not have that same caution when it came to my infrastructure. Their database service became “generally available” less than a year ago! To their credit, DigitalOcean has been fairly aggressive in the past couple of years to add and improve their non-droplet services. Maybe in a year DigitalOcean’s newer services will become more reliable, but I don’t have a year to wait.

Back to AWS’ well-monied arms I went.

Sturdy Ships

Getting everything running again on AWS cost me a weekend. I was already familiar with it and nothing about my DigitalOcean infrastructure was proprietary. For example, all I needed to do on Packer is change the build target from DigitalOcean to AWS and a few small script changes. Using non-proprietary technology (e.g. Terraform, Packer) gives you an exit strategy for free. Generally having an exit or alternative strategy whenever you are exploring something unknown is a good idea.

Using technology you already know (in this case, AWS) is a huge time saver and even though DigitalOcean is cheaper, at my current stage of my life my time is more expensive.

Admittedly, I think what made me truly stray away from AWS was the thrill of learning something new. The dark truth was that I didn’t really have to use DigitalOcean. It is pretty to find yourself with a few thousand dollars of AWS credits for a couple of years if you know where to look. Plus, it’s still possible to learn new things with technology you already know! When I migrated to AWS I decided to try setting up an ALB for the first time and bam, I learned something new in a smaller, safer way!

In hindsight, I should have started on AWS and only transitioned to DigitalOcean once cost became an issue. It was a classic case of premature optimization.