Table of Contents
Changing how and where you host your product is one of the most challenging decisions for a saas company to make.
When you first get started, effectively any hosting setup will do the trick: your application is small, you have relatively few customers, and perhaps (depending on your sales process) expectations around performance and reliability are a bit lower during the beta phase.
Over time all of that changes. Your application grows more complex, your customers 10x, then 100x, which means the amount of data you’re storing and moving around grows by 1000x. At these orders of magnitude, even the most well written product begins to fail – and most of the time the V1 wasn’t well written.
All of a sudden (it feels) customers are frustrated with slow performance and low reliability, you’re not able to meet your customers’ needs as they grow, and the complexity of how your app is hosted and deployed creates a dampening effect on how quickly you can ship new features.
In the early days you’re able to ship a feature a day and all of your customers are thrilled… now it takes a day for a change to get through CI!
Eventually the question comes up: Should we rethink how our product is hosted? Like… from the ground up? Companies generally avoid making this decision for a few big reasons:
- It’s expensive. Like, really expensive.
- It distracts the team from writing new features + fixing bugs.
- It requires a change in skill set (and a specialized skill set to do the migration.)
- You have to face challenging tradeoffs, the more of your team’s time you buy with hosted services, the less customization and higher cost you incur.
- And… It’s not fun. None of us started a saas company to deal with server migrations. Unless you’re like Reilly Chase from Hostifi (sorry Reilly.)
But eventually you need to decide what stage your business is in. Are you growing and want to scale? Do you think that there are 10x or 100x opportunities for your business ahead?
If you want to scale, you need to make the investment of time and money and make sure your infrastructure suits your size of business.
And that, my friends, is how we spent $400k extra in hosting over the past 4 months.
Lessons Learned
You might be wondering how you could pseudo-accidently spend $400k to a hosting provider in such a short period of time. The good news is that it’s surprisingly easy to do!
Wait… that’s bad news. Here’s a few lessons we learned along the way so you don’t follow exactly in our footsteps (but we do recommend you address your hosting issues eventually.)
- Have a plan
- Decide how much DevOps you want in your development
- Talk to who you’re going to switch to first
- Make sure you have the right team in place
- Brace for transition period costs
- Plan for higher Cost of Revenue
Have a plan
The first mistake we made was taking the brute force approach. We resolved to make the change we wanted, and made the decision to get through it as soon as possible.
And, we thought that the fastest way through was getting hands on keyboard ASAP. I’ll go into more detail more below, but essentially there’s a lot you can do to set yourself up for success (i.e., spending less than $400k) before you even get started.
I think a lot about psychology in my day to day of running a business, and in retrospect the “brute force” approach feels a bit like the coping technique of closing your eyes and nose to take some medicine you don’t like the taste of.
My advice here is to soak in the pain up front and sit with it to save a lot more pain (and money, have I mentioned that yet?) down the road.
Decide how much DevOps you want in your Development
Deciding up front how much devops you want your dev team to do is a critical step. From my experience, the three options are usually:
- I don’t want my engineers to even be able to pronounce Kubernetes
- I want my engineers to understand how to manipulate pipelines and automation
- I want my engineers to understand the whole stack and use DevOps as a way to deliver features.
I’m personally in camp one, but a lot has changed in the world of DevOps since I started my last business. Platforms are easier to use and better built than ever, and there’s now a lot more options for how hosted you want your app to be.
When I started my first business your options were something like buy a server and pay someone to physically take care of it, or, go totally hands off with Heroku. Now there’s a solution for pretty much every step along the way.
There’s no global right or wrong answer here, just what’s right for your company and team. The more DevOps skills you want your team to have, the fewer engineers will show up with the exact combination of skills you want. The less DevOps your team has control over, the fewer levers they have to impact performance, and again, some features we deploy can be owned by DevOps and not writing our own software.
Getting a good sense of where you want to fall in this spectrum up front makes sure you’re evaluating the right set of options, and don’t lose your way. You’ll find a lower cost along the way if you want your team to do more work, just make sure when you get to the end you satisfy the reason for starting.
Talk to who you’re going to switch to first
This is an obvious one in retrospect: before you write even the first line of code or kick off the first command to switch hosting providers, talk to who you’re going to switch to first.
Of course, our “brute force” approach didn’t include this step.
A lot of our workload was getting moved to AWS, and wouldn’t you know it: they want you to move compute to them so they have migration services to match.
They can do everything from helping you plan out the move, forecast costs during and after the migration, and connect you with experts that can take over some of the detailed work you’d only know if you moved apps to their platform as a job.
This should also be a consideration as you’re making your move. It’d be totally reasonable to pick a new hosting partner based on how quickly and painlessly you can make the move.
Make sure you have the right team in place
One of the challenges with moving your hosting provider is that you’re touching every part of your product. Not only are there domain specific skills to have, but you also need to involve teammates who deeply understand all parts of your product since a proper migration is going to require at least a few changes to the codebase.
For example, since we’re switching from bare metal hosting we quickly learned that the volume of data getting transferred between our services was going to be both too slow and too expensive to maintain. This is a fundamental change to how the application works, and fortunately this was already something we had our eye on optimizing—making it much more expensive just bumped our priority up.
As we were working through our migration we realized we didn’t have the team we needed and hired two new senior developers to our team, as well as borrowing a few migration experts from our networks.
The likelihood that you move your hosting provider without needing to rewrite at least a few parts of your product are really unlikely, don’t get caught making the mistake of “we’ll keep our dev team focused on their backlog and elect a separate team to do the migration.” This just leads to that other team bugging the dev team to make code changes and then no one is happy.
Also, per the point above, think about reaching out to a team that has expertise wherever you’re weak. We’ve been working with DoIT International who have a particular expertise in getting your servers right sized and finding other savings opportunities (at no speed or performance cost.) You don’t need to work with them, but, I’d recommend tapping a set of experts, it’ll pay for itself.
Brace for transition period costs
During the transition period you’ll both be transferring large amounts of data and paying for multiple hosting environments.
This cost can be mitigated in your planning stage, and by working with your new hosting provider. This is a really simple (but expensive) detail to miss.
Keep in mind that when migrating to a new infrastructure, it’s not a flick of a switch. For long periods of time you need both the old and new infrastructure running so you can test the success of your new infrastructure, and not have a drop in quality that impacts customers.
We ended up spending a lot of money during this phase, which is necessary, but very painful. Especially when we had to make changes and then transfer data again, incurring the transfer costs again.
Other than working on a data transfer deal with your host I don’t have any better advice than to mark it in your budget and hope.
Plan for higher Cost of Revenue.
Now that we’re finally over into our new host everything is cheaper, right?!
Well… maybe! But not for us.
We took a lot of the DevOps responsibilities off of our development team so that our current developers, and all the new ones we hire, can spend more time focusing on writing software. That means that we’re paying more to hit the same baseline, and even in our best forecast hosting costs will be 50% higher than before.
Long term costs are an important part of the plan to make sure that this change doesn’t cause cash flow problems. Hopefully you’ve done your diligence and have made the move as quickly and cost effectively as possible, now it’s time to make sure that 6 and 12 months down the road we’re still in good financial standing.
The other important finance note (especially if you’re reporting to a board or investors) is that you could be moving an operating expense into cost of revenue, which generally means very little to your average SaaS operator, but it means a lot to the people who write us checks :).
Ok, so, was it worth it?
After spending a few thousand words explaining these trials and tribulations you might expect that I’d be saying the migration wasn’t worth it. And if there was an easier way out, I would have taken it. But, in reality these costs and challenges are sort of par for the course. If we weren’t Baremetrics, and in the habit of sharing every time we learned a painful lesson, we would have just run with it.
As I mentioned right in the beginning, the key question is what stage is the business in? I’ve been with Baremetrics for the past year and have seen the team level up a few times, and there’s huge opportunities that we’re just now getting a chance to go after. We needed to do this sooner or later, and we decided to dive in (perhaps with a bit too much haste) and get this done ASAP so we can focus on the future.
I’ve mentioned a few times throughout, but here’s the summary of why I’m excited we did this:
- More time for features and performance, and less time on configuration and maintenance
- Greater concentration of tools, meaning we can get more people to a higher level on fewer pieces of tech
- More control (surprisingly) on the key areas the impact our performance (YMMV)
- Better uptime and reliability for more regions.
Money well spent. Future, here we come.
I’d be remiss if I didn’t point out the heroic effort of our team in getting this migration across the line. We’ve doubled our dev team over the course of the project, and almost everyone will be hanging around after to keep building.
We definitely didn’t pick the cheapest option, but we picked the option that gives us the most firepower going into the next 10 years.
And we’re just getting started—this move represents a new foundation for us, and we’re excited to keep building up.
Reach out to us!
We’re stoked to have you in our community! If you ever need anything, reach out to us at hello@baremetrics.com and let us know. Our team is here to help you connect data sources, build dashboards, and make sure you make more from your data.
