Vertical Scaling: Bigger Hardware
The simplest response to a struggling server is to make it bigger. More CPU cores, more RAM, faster SSDs. This is vertical scaling, scaling up. No code changes required. No distributed systems complexity. Just buy a larger instance.
Vertical scaling is like replacing a 2-lane road with an 8-lane highway. It's faster, but you're still building on the same piece of land, there are hard limits to how wide you can go.
When vertical scaling wins
Simple and fast
No application changes. Resize the instance, restart, done. The fastest path from 'slow' to 'fast'.
No distributed complexity
One server means no network partitions, no consensus, no data synchronization headaches.
Single-threaded workloads
Some databases (older MySQL configs) or runtimes (Python GIL) benefit more from faster CPUs than more cores.
Stateful applications
If your app stores state in memory (sessions, caches), vertical scaling avoids the complexity of sharing that state across servers.
The hard limits of vertical scaling
Vertical scaling has a ceiling. AWS's largest instance (u-24tb1.metal) has 448 vCPUs and 24 TB of RAM, but it costs ~$250/hour and there's exactly one size beyond it: nothing. More practically, the ceiling you hit first is usually cost, not physical limits.
| Problem | Why vertical scaling doesn't solve it |
|---|---|
| Single point of failure | One server equals one failure domain. It goes down, everything goes down. No amount of RAM prevents hardware failure. |
| Geographic latency | A 96-core server in Virginia is still 70ms from London. Physics doesn't care about your CPU count. |
| Cost efficiency at scale | Doubling CPU doesn't double throughput. After a point, more servers (horizontal) is cheaper than bigger servers. |
| Maintenance windows | Rebooting a single giant server takes everything offline. Multiple smaller servers can be rolled one at a time. |
Don't skip vertical scaling prematurely.Many startups jump to microservices and Kubernetes before they've proven product-market fit. A single well-tuned server can handle more than you think, Instagram served 30M users from 3 servers before their first major rewrite.
Horizontal Scaling: More Servers
Instead of one big server, use many smaller ones. Horizontal scaling, or scaling out, adds capacity by adding instances. Each server handles a fraction of the traffic. Together, they handle the whole.
Horizontal scaling is like building more lanes on more roads in parallel cities, rather than making one road infinitely wide. It trades single-point-of-failure risk for distributed complexity.
The stateless requirement
Horizontal scaling only works when any server can handle any request. This requires your application servers to be stateless, they don't store anything in memory between requests. All state lives in shared, external systems.
Auto-scaling: elastic capacity
Cloud platforms can automatically add or remove servers based on load metrics. This is auto-scaling, you define the rules, the platform adjusts capacity. This means you pay for what you use and handle traffic spikes without manual intervention.
Common for CPU-bound workloads (image processing, ML inference)
For async worker pools, keep queue shallow
Scale before the spike (e.g., before a product launch)
Scale in to save cost during quiet periods
The database bottleneck: Horizontal scaling is straightforward for stateless app servers, but databases are harder. A single primary DB can become the bottleneck. Solutions: read replicas for read-heavy workloads, connection pooling (PgBouncer), sharding for write-heavy workloads, or switching to databases designed for distribution (Cassandra, DynamoDB).
Load Balancer Algorithms
A load balancer is only as good as its distribution algorithm. The right choice depends on whether your servers are homogeneous, whether requests have varying costs, and whether users need to be pinned to a specific server.
Routes requests in sequence: Server 1, Server 2, Server 3, Server 1... Equal distribution by count, not by load. Simple and effective when requests have similar cost.
Routes to the server with the fewest active connections. Naturally adapts to servers handling slow requests, faster servers accept more connections as they complete work.
Hashes the client's IP to consistently route them to the same server. A poor man's sticky sessions, no cookie required. Breaks if server count changes (consistent hashing fixes this).
Like Round Robin, but servers with higher weight receive proportionally more requests. Useful when servers have different hardware specs, a 16-core server gets 2x the traffic of an 8-core one.
Picks a server at random. Surprisingly effective at scale, by the law of large numbers, distribution converges to even. Simpler to implement than Round Robin with similar results.
See it in action: the simulator
The simulator below models a round-robin load balancer with equal capacity servers. Drag the traffic slider to push the system toward overload, then add servers to restore health. Notice how response time degrades non-linearly as utilization approaches 100%.
Queueing theory: At 50% utilization, response time ≈ 2× base. At 90%, it's 10×. At 99%, ∞, the queue never drains.
Try pushing to 120 RPS on 1 server, then add servers to watch the response time recover.
The queueing model above (M/M/1) is a simplification, but the intuition is correct. Real systems degrade faster due to garbage collection pauses, memory pressure, and lock contention. Keep production utilization below 60-70% to maintain headroom for traffic spikes.
Real-World Scaling Patterns
Every high-scale system uses a combination of techniques rather than a single scaling strategy. Here are the patterns that appear repeatedly across large-scale architectures.
Database read replicas
Most applications read far more than they write (read:write ratio often 10:1 or higher). Horizontal scaling the app tier is easy, but the database becomes a chokepoint because every server hits the same primary. Read replicas solve the read side.
Writes only
Replica 1
Reads
Replica 2
Reads
Replica 3
Reads
App servers route writes to primary
App servers route reads to any replica
Replicas lag primary by milliseconds (async replication)
Replication lag trap:Replicas are eventually consistent with the primary. If a user writes a record and immediately reads it, they might hit a stale replica that hasn't received the update yet. For reads that must be fresh after a write, route to the primary or use "read-your-own-writes" consistency with a short primary bypass window.
The caching layer: skip the database entirely
The most effective scaling technique is not doing work at all. If a response can be served from cache, the database never sees the request. Redis or Memcached sitting between your app servers and database can absorb 90%+ of read traffic for read-heavy workloads.
App checks cache first. Miss? Read from DB, write to cache. Most common pattern. App code is cache-aware.
General-purposeEvery write goes to DB and cache simultaneously. Cache is always warm. Write latency doubles.
Read-heavy, tolerate write latencyWrite to cache immediately, async-flush to DB. Fast writes, risk of data loss if cache node dies.
Write-heavy, can tolerate stalenessSharding: horizontal database scaling
When a single database (even with replicas) can't handle write volume, you split data across multiple independent databases, each owning a shard of the keyspace.
| Shard key | Example | Risk |
|---|---|---|
| User ID | user_id % 4 -> Shard 0-3 | Hot users (celebrities) create hot shards |
| Geographic region | EU -> Shard A, US -> Shard B | Uneven growth; US might outgrow EU 3x |
| Consistent hash | hash(user_id) mod N | Complex rebalancing when adding shards |
| Date range | 2024 data -> Shard C, 2025 -> Shard D | Recent shards are always hottest (writes) |
Sharding is a last resort. It adds massive complexity: cross-shard queries are expensive, transactions spanning shards require distributed coordination, and rebalancing is painful. Exhaust read replicas, connection pooling, caching, and vertical scaling before reaching for sharding. Most applications never need it.