Note: Unless otherwise stated, this post describes how a MongoDB replica set behaves by default. If you want a more complete understanding of how you can tweak these behaviors, you can read the documentation for Read Preferences, Read Concerns, Write Concerns, and Sharding. Oh, and probably Causal Consistency as well.
Now, on to the post.
Some years ago, I worked at a company that used MongoDB, hosted on Atlas1. We had a multi-node replica set, so we thought we had horizontal scaling covered.
But when I explored our dashboard in Atlas, that’s not what I saw. Only the primary node was receiving traffic. The secondary nodes were acting as nothing more than hot backups. And that’s when I began my education in how MongoDB reads and writes actually work.
By default, reads and writes both go to the primary node. This buys you a certain level of high availability - if one node fails, another one gets elected, and you keep moving.
What it doesn’t buy you is horizontal scaling. If you’re running a replica set using the default configuration, you’re paying for hot swappable nodes. That’s it.
So, what does it take to scale horizontally in MongoDB?
Well, for writes, your only option is to switch to a sharded cluster. Happily, sharding can also scale your reads. Sharding is beyond the scope of this post, but let’s just say that it’s not a trivial decision. If you don’t have a sharded cluster, then you can only scale writes by growing the server (aka vertical scaling). As I’m writing this, every time you go up one server size in Atlas, it roughly doubles your operational costs.
But maybe your pattern is more read-heavy. For reads, you have more options. Your callers can set Read Preference so that they can read from secondary nodes. They just have to accept a little more “eventual” in their eventual consistency. Which could be fine if they’re designed to tolerate that. But if they expect to, say, read their own writes to capture an autogenerated ID, that may not work out so well. Then again, with the default Read and Write Concerns, that wasn’t guaranteed in the first place. But … you knew that, right?2
Because Read Preference is set by the caller, changing it means modifying the service’s MongoDB driver settings, whether programmatically or by changing the connection string. So first you have to identify which services can tolerate reading from a non-primary node, and then you may need to update the code before you can deploy it. This does not make for a stress-free Friday afternoon. And that’s not even getting into the need to run your MongoDB queries within sessions, not to mention how Read and Write Concerns interact with Read Preference.
No distributed database is simple. As soon as you’re running more than a trivial load, someone in your org needs to be willing to do the research, make decisions, and set developer standards for interacting with shared resources. If you don’t, best case, you’re probably wasting money. Worst case, you’re setting yourself up for a Real Bad Day when your site suddenly goes viral.
Atlas is MongoDB’s managed hosting service. It’s actually pretty slick, but everything I’m saying about MongoDB in this post applies regardless of whether you’re using Atlas or running it yourself. ↩︎
In order to Read Your Writes, three things need to be true: Your code must explicitly be opening sessions, your Read Concern must be “majority”, and your Write Concern must be “majority.” You’ll need to inspect your code to see if you’re using sessions. The default Write Concern is “majority”, but the default Read Concern is “local”. So if you want to safely read your writes, you need to be careful in your coding and you need to change the defaults - which of course has its own tradeoffs. ↩︎