Enter →
Understand how systems actually work, so you direct the build instead of prompting blind.
The machines can write the code now. That changes what’s worth learning. Your leverage is no longer in typing the system, it’s in understanding it well enough to design it, brief it, and judge what comes back. This manual builds that understanding from the ground up, starting with the one server you’ve already deployed and growing it, decision by decision, until planetary-scale systems make sense.
Understand, don’t memorize
Architectures are outputs. What transfers is the reasoning that produced them, the forces, the trade-offs, the “why.”
Play, don’t just read
The hard-to-feel ideas, load breaking a server, the latency ladder, tail-at-scale, are live instruments here. Operate them.
Direct the build
Every chapter ends by turning understanding into leverage: how to brief an AI to build the thing, and how to tell if it got it wrong.
It doesn’t teach you to write code, the AI does that. It teaches you to be the person who knows what should be built and why. Where an old textbook would hand you pseudocode, this one hands you a Directing the Build panel: the architectural brief you’d give, and the specific things you’d check in what the AI returns. That is the difference between a vibe coder and an architect.
How I got here, from instinct to method
I didn’t learn any of this from a textbook. I learned it by building things that worked, and then building things that broke.
For a long time I built on pure instinct. If it ran, it shipped. I had good intuition and no vocabulary, I couldn’t have told you why something held up or when it would fall over, only that it did or it didn’t. That is the vibe coder’s superpower and its ceiling: you can make almost anything, right up until it meets real load, a real bill, or a real attacker.
The walls came one at a time, and always as a surprise. A server that ran fine for months and then simply fell over. Heavy work that should never have been on the request path. A cost that quietly tripled. A security review that found soft spots I’d never thought to look for. Each time, I fixed it after it hurt, reacting, patching, moving on.
Then the pattern showed itself. The failures weren’t random; they were the same few shapes over and over. The single box that couldn’t take more. The boundary that shouldn’t have been crossed. The estimate I never did. Once I could see the shapes, I could see them coming, and that changed everything. Building stopped being “does it run?” and became “does it hold up?” I started estimating before I built, chasing the bottleneck on purpose, designing for the failure instead of hoping against it, and directing my tools instead of being surprised by them.
That shift, from reacting to anticipating, from instinct to method, is the whole distance from vibe coder to architect. It took me years of getting surprised. This manual is my attempt to compress it so it takes you a good deal less. The From the field notes scattered through the chapters are the real moments where my thinking had to change, each one a wall I hit, so that ideally you will see yours coming.
How to use it
Read in order, at least once. It’s a staircase, not a reference shelf, each chapter leans on the one before. Operate every instrument. Move the sliders until the numbers surprise you; that surprise is the lesson landing. Do the practice. System design is a reasoning skill, and you only build it by reasoning. Reveal the solutions after you’ve tried.
Start where you already are: with something you’ve built.
About the author
A working architect, two decades of systems that had to hold up.
Konrad Voss is a systems architect who has spent the better part of two decades building software that has to stay up, under real traffic, real budgets, and real failure.
He has designed and scaled production systems from the first single-server prototype to platforms carrying millions of requests a day, and, just as often, been the one called in to rescue systems buckling under load they were never built for. Backend architecture, data pipelines, caching and storage layers, the migrations that have to happen with users still online: this is the terrain he works in, and has for a long time.
Everything in this manual was earned in production, not a lecture hall. The outages, the runaway bills, the security audits, the 3 a.m. pages, those were the curriculum, and the From the field notes throughout the book come straight from systems he built and, often enough, watched break. It is the book he wishes he’d had at the start: the reasoning behind the decisions, not just the decisions.
His conviction is simple. Now that a machine can write the code, the scarce and valuable skill is understanding a system deeply enough to design it, direct the build, and judge what comes back. That is what this manual sets out to give you, not a set of answers to memorize, but the way of thinking that produces them.
Enough about the author. The systems are the story.
What System Design Actually Is
Why building and designing are different skills, and why the second one is now your real edge.
- Why building software and designing systems are different skills
- The two kinds of requirements, and why beginners only ever see one
- The three forces every system balances
- The mindset shift this whole manual is really about
You’ve already done this
Every time you put images in cloud storage instead of your database, added a cache because a page felt slow, or split your code into a separate API, that was system design. You were deciding how the pieces fit together to meet a goal under real constraints. You did it on instinct. This manual makes it deliberate.
System design is the practice of deciding how the components of a software system fit together so it meets its requirements under real-world constraints like traffic, money, and failure.
The difference between a builder and a designer isn’t secret technology. It’s that the designer sees the decision coming, names the options, predicts what each costs, and chooses on purpose instead of reacting after something breaks.
Think of a time something you built slowed down or fell over under load. You almost certainly fixed it reactively, noticed the pain, then patched it. Design is learning to see that pain before it arrives. Same moves, better timing. And in the AI era, that foresight is exactly what you can’t outsource, the model will build whatever you ask, including the wrong thing, quickly.
Building vs designing
Building answers one question: does it work? You run it, it does the thing, you ship. Designing answers a harder one: does it hold up? Does it survive 100× the users? A server dying at 3 a.m.? A tripled bill? Two people buying the last item in the same millisecond? Most software that “works” would fall apart under those questions, not because it was built badly, but because it was never designed for them.
“It works for my current users” tells you almost nothing about 100× your users. Systems don’t degrade smoothly, they hit a wall. A database happily serving 1,000 requests/sec can collapse at 1,200. Not slow by 20%, collapse. Seeing the wall before you hit it is the point.
The two kinds of requirements
Functional requirements are what the system does, the features. “Users can post a photo.” These are visible, so you think about them naturally. Non-functional requirements are how well it does them under pressure, and they’re where design lives: scale, latency, availability, consistency, durability, cost. They’re invisible until they’re violated.
Functional requirements decide what you build. Non-functional requirements decide how you build it. A note app for 500 people and one for 500 million can have identical features and be completely different systems, because their non-functional requirements are worlds apart.
The three forces
Nearly every design decision is a negotiation between three forces pulling against each other, and scale is the pressure that stretches all three at once.
Figure 1.1. Pull harder toward any corner and you usually give something up at the others. There is no perfect design, only appropriate trade-offs.
Performance (is it fast?) usually costs money or complexity. Reliability (does it stay up and keep data safe?) almost always costs money, you pay for spare capacity you hope never to use. Cost, the cheapest system is one slow server with no backups, which is to say cheapness sacrifices the other two corners. At ten users none of this bites; at ten million, every corner is under tension at once, and the balancing act is the job.
Beginners fear building something too weak. But building for Netflix scale when you have 200 users is its own expensive mistake, months of complexity you don’t need, slowing down the thing that matters early: shipping and learning. Good design matches the architecture to the actual forces, not the imagined ones. This matters double when an AI will happily generate a sprawling microservice architecture the moment you hint at “scalable.”
The mindset shift
If this chapter installs one thing, it’s a second question that now runs alongside your old one. As a builder you asked: does it work? From now on, beside it runs: does it hold up, at scale, under failure, within budget? Everything else in this manual is just the accumulated set of standard answers to that second question. You don’t memorize them as trivia; you learn to feel the pressure each one relieves. That’s why the next chapter doesn’t start with a component, it starts with the app you’ve already built, and puts it under load until you can feel exactly where it breaks.
Here’s the payoff for the AI era. When you ask a model to “build me a photo app,” it will invent the non-functional requirements for you, and it will usually guess wrong, either over-building or ignoring the constraint that actually matters. The architect’s move is to state the non-functional requirements yourself: “roughly 10k daily users, read-heavy, images up to 5 MB, cost-sensitive, occasional downtime is fine.” Those sentences shape the architecture far more than the feature list. By the end of Part I you’ll be able to write them from numbers, not vibes.
- System design
- Deciding how components fit together to meet requirements under real constraints.
- Functional requirement
- What the system does, its features.
- Non-functional requirement
- How well it does it under pressure: scale, latency, availability, consistency, durability, cost.
- Trade-off
- Giving up something at one corner to gain at another, the essence of design.
Pick an app you use daily. Write three functional and three non-functional requirements. For each non-functional one, guess whether it must be strict or loose (does a bank transfer need strong consistency? does a “last seen” timestamp?).
No single right answer, the skill is the split. WhatsApp example: Functional, send a message, see read receipts, make a call. Non-functional, messages feel instant (strict latency), the app is almost never down (strict availability), “last seen” can lag a few seconds (loose consistency is fine). Notice how the strictness differs, a dropped message is unacceptable, a stale “last seen” is invisible. That difference is the insight.
Name which force each helps and which it hurts: (a) a second copy of your database on another machine; (b) a cache in front of your database; (c) one cheap server, no backup.
(a) Helps reliability (survives a failure) and can help performance (spread reads); hurts cost (two machines) and adds sync complexity. (b) Helps performance and can cut cost (fewer DB queries); risks stale data and adds complexity. (c) Helps cost (cheapest); hurts reliability (one failure loses everything) and performance. The classic “cheap corner.”
We take the single-server app you’ve deployed, put it under growing load, and watch exactly where it breaks, then make the first four moves every system makes.
The One-Server App (and Why It Breaks)
Start from what you’ve deployed. Put it under load. Watch the walls appear, and the moves that push them back.
- The architecture you’ve almost certainly already deployed
- What physically happens as more people use it
- The first four moves every growing system makes, in order
- The single most important concept here: stateless
- Vertical vs horizontal scaling, and how to find a bottleneck
The app you’ve already built
A frontend people open, a backend that runs your logic, a database that stores the data, all on one server somewhere. It works. There is nothing wrong with this. Every large system in the world began roughly here. The goal isn’t to shame the single server; it’s to feel exactly when to leave it, so you neither cling too long nor abandon it too early.
So let’s put it under pressure, and rather than describe it, operate it. Step through the moves below and watch the architecture assemble.
What actually breaks
Your server has a fixed amount of four things: CPU, memory, disk, and network, and your app and database compete for all four on the same box. As traffic climbs, CPU saturates and requests queue; memory fills and the machine starts swapping to disk (roughly a thousand times slower · Chapter 4 shows you why); and the app and database starve each other. Then the killer: it’s one machine. If it dies, the whole product is down, and if the disk held the only copy, the data may be gone. That’s a single point of failure.
The single server doesn’t slow down and warn you. It runs fine, fine, fine, then hits a wall and falls over. Scaling is the art of adding capacity and removing single points of failure before you reach that wall.
Years ago I built a data-heavy web app and ran the whole thing on a single small instance, 512 MB of memory, backend and all. Ordinary requests were fine. Then a user triggered a broad query: the backend pulled in a batch of documents, spun up several in-memory data frames, and ran a couple of models over them, all in one process. Memory shot past the limit and the API didn’t slow down, it just shit the bed: a 502 straight to the user. Fine, fine, fine, then a wall, exactly as above.
The fix wasn’t a bigger dial. It was seeing that one box was doing too much synchronous work on the request path: I moved repeated work off it, computing only novel results and caching the rest, and eventually gave the heavy lifting its own machine.
⚠ If this breaks
This is the whole motivation for the book in three boxes. With exactly one of everything, the server and the database are each a single point of failure: lose either and the product is gone, and if that one disk held the only copy, so is the data. Every move in the chapters ahead, another server, a replica, a cache, a backup, exists to take one more box off this list. You just felt the problem; the rest is the cure.
The most important concept: stateless
Look at Move 2 in the stepper, a load balancer spreading traffic across several app servers. For that to work, any server must be able to handle any request. So no server can secretly store something the others don’t have.
The trap: suppose you keep each logged-in user’s session in the memory of the server that handled their login. Next request lands on a different server that’s never heard of them, suddenly they’re logged out. The server was holding state, and state is the enemy of horizontal scaling.
A stateless server keeps nothing important on itself between requests. Everything that must be remembered, sessions, data, uploaded files, lives in a shared place all servers reach. A stateless server is interchangeable, like a cashier at a well-run store: any one can serve any customer because the information lives in the shared till system, not in one cashier’s head. Make servers stateless and horizontal scaling just works, add one and the load balancer uses it; kill one and it routes around it; no user notices either event.
Ever deployed to Vercel, Railway, Render, or a serverless setup and it “just scaled”? That’s because those platforms force your app to be stateless and run many copies behind a load balancer for you. You’ve been enjoying this principle without naming it, and it’s exactly why writing to a local file or storing a session in memory mysteriously breaks on those platforms.
Two words you now own
Vertical scaling (scale up), a bigger machine. Dead simple, no code changes, but a hard ceiling, wildly expensive at the top, and still a single point of failure. Horizontal scaling (scale out), more machines. Nearly unlimited and removes single points of failure, but demands statelessness and the coordination challenges that fill the rest of this manual. Reach for up first because it’s easy and buys time; reach for out when you need to pass one machine’s ceiling or can’t tolerate a single point of failure.
A system is only as strong as its current bottleneck. Adding capacity anywhere else does nothing. The whole job is a loop: find the wall, push it back, then find the new wall that relieving the first one exposed. Notice the evolved system in the stepper still has one lonely database, we relieved the app tier, so the database became the next bottleneck. That’s not a flaw in the diagram; it’s the job.
Two things you now know to insist on when an AI scaffolds a backend. First: “keep the app servers stateless, sessions and uploads go to Redis and object storage, never local memory or local disk.” Models routinely default to in-memory sessions or saving files to ./uploads, which works on one box and shatters the moment you scale out. Second, when you review generated infra: ask where state lives. If the answer is “on the server,” you’ve found the bug before it ships. That single question is worth more than reading every line of the code.
- Single point of failure (SPOF)
- A component whose failure takes down the whole system.
- Load balancer
- Spreads incoming requests across multiple servers behind it.
- Stateless
- A server that keeps nothing important on itself between requests; shared state lives in a common store. The key enabler of horizontal scaling.
- Horizontal / vertical scaling
- Adding more machines / making a machine bigger.
- Bottleneck
- The single most constrained component, which caps the whole system’s capacity.
- Source of truth
- The authoritative store for a piece of data (usually the database), as opposed to caches or copies.
Name the likely bottleneck and the move that relieves it: (a) app + DB share one box, CPU pegged at 100%; (b) ten app servers, but the same few DB rows are read on every request and the DB is overwhelmed; (c) users in Asia are slow, but US servers are idle, the slow part is images; (d) one powerful server, fast, but it crashed and you were down three hours.
(a) App and DB contending, Move 1, separate the database. (b) The database hit for hot data, Move 3, add a cache. (c) Distance, Move 4, a CDN near those users. (d) A single point of failure, go horizontal (Move 2). Note a bigger server wouldn’t have helped: the problem wasn’t capacity, it was having only one of something.
For each, does storing it locally on the server break horizontal scaling, and where should it live? (a) a login session; (b) a temporary variable used only during one request; (c) a user’s uploaded photo saved to local disk.
(a) Breaks it, put it in a shared cache (Redis) or the DB. (b) Fine, it never outlives the request, so no other server needs it. Stateless means “nothing important kept between requests,” not “no variables.” (c) Breaks it, the photo is invisible to other servers; it belongs in shared object storage.
Which of the four moves has your project already made? Which would you make next, and what specific pain would it relieve? Three sentences.
Typical honest answer: “DB is already separate (Move 1). Still one app server, so it’s a single point of failure, no Move 2 yet. Next I’d add a load balancer and a second app server, so one crash doesn’t take me down and I roughly double capacity.” If you’re on Vercel/Render you may have Moves 2 and part of 4 for free, note what local state or local file storage would still break.
Before adding any more machinery, we learn to estimate, and you’ll operate a console that turns “a simple photo app” into the numbers that force an architecture.
Thinking in Numbers
Rough arithmetic that tells you which architecture you need, before you build a thing.
- Why a rough number beats a precise guess
- The few figures worth memorizing
- A repeatable method: from users to servers and storage
- An estimation console you operate, watch the numbers force the design
Why estimate at all
A note app for your book club and a photo app for ten million people both “store data and serve pages,” yet one is happy on a single small server forever and the other needs petabytes of storage and its own CDN. The only thing that tells them apart is numbers, and you don’t need exact ones. You need to be right within a factor of ten, because that’s the granularity at which architecture decisions actually flip. 200 requests/sec vs 250 changes nothing; 200 vs 200,000 changes everything.
Back-of-envelope estimation is rough arithmetic to find a system’s order of magnitude, the right power of ten, so you can tell which architecture it needs. You should be able to do it on the back of an envelope, in two minutes, no calculator.
The figures to memorize
Data sizes climb by 1,000s: KB → MB → GB → TB → PB → EB. Useful anchors: a short text post ≈ a few hundred bytes; a phone photo ≈ 2 MB; a minute of video ≈ 10–50 MB. And the one time conversion that does most of the work: seconds in a day ≈ 100,000 (it’s 86,400, round it).
Don’t reach for a calculator. If estimation starts to feel like real math, you’re doing it wrong, round aggressively and keep moving. “About 2.5 million,” not 2,592,000.
The method, as a machine
Estimation flows one direction: people → requests → machines and storage. Rather than read the steps, drive them. Move the sliders and watch three things update: how much compute you need, what kind of storage, and whether you need a CDN. The verdict panel lights up the components the numbers demand.
Try it: start at the defaults (a photo app, 10 M users) and watch it demand object storage and a CDN. Now drag users down to a few thousand and item size to a tiny text post, the verdict flips to “a single server is genuinely fine.” The math designs the system.
At the photo-app defaults, a “simple” app needs petabytes a year (no normal database holds that, you need object storage) and tens of gigabytes per second of egress (which would bankrupt a single origin, you need a CDN). You haven’t drawn a box, and the numbers already told you the architecture. That’s the entire point: estimation isn’t busywork before the design, it is the first act of design.
In hindsight that crash was arithmetic I never did. A single broad query held a batch of documents, several data frames, and a model’s working set in memory at once, call it a few hundred megabytes for one request. On a 512 MB box, two or three concurrent broad queries is already an out-of-memory crash. Nobody had multiplied memory per request × concurrent requests against the box’s ceiling.
Most systems read far more than they write, a feed is read a thousand times per post. This read/write ratio shapes everything: a read-heavy system leans on caches and read replicas (cheap to add), while a write-heavy one stresses the database’s write path (genuinely hard, that’s Part III). Watch the console: pushing reads up lights the cache; pushing writes and size up lights storage. Different pressures, different architecture.
This is the single highest-leverage habit in AI-assisted building. Before you ask a model to build anything, do the two-minute estimate and hand it the numbers as constraints: “~10k daily users, 50:1 read/write, images up to 5 MB, so assume object storage for media, a CDN in front, and a cache for hot reads; a single Postgres is fine for metadata.” That paragraph steers the whole architecture. Without it, the model picks defaults blind, often a fleet of microservices you don’t need, or a naive design that stores 5 MB images as database blobs. Your estimate is the most important part of the prompt.
- Order of magnitude
- The power of ten a number sits at, the granularity at which architecture decisions change.
- Daily active users (DAU)
- People who use the system on a given day, not total sign-ups.
- Peak factor
- How much higher peak traffic runs than average (often 2–10×). You size for the peak.
- Read/write ratio
- Reads per write; decides whether you lean on caches/replicas or on the hard write path.
- Egress
- Data flowing out to users per second; large values force a CDN.
100 M new links/day, each clicked ~100 times, each stored record ~500 bytes. Using the console (set item size to its smallest and reason about the rest): what are reads/sec vs writes/sec, the read/write ratio, and roughly the storage/year? Which components matter most?
Writes ≈ 100M ÷ 100,000 = ~1,000/sec. Reads ≈ 10B clicks ÷ 100,000 = ~100,000/sec. Ratio ≈ 100:1, extremely read-heavy → the cache matters most (hot links served from memory), plus a load-balanced app tier for 100k req/sec. Storage ≈ 100M × 500 B × 365 ≈ 18 TB/year, fits a normal database, so no exotic storage. And responses are tiny, so a CDN matters far less than for the photo app. The numbers cleanly rank your priorities.
Estimate one of your projects: DAU (even 50 is fine), actions/user, requests/sec, storage/year. At those real numbers, is a single server actually fine?
For many real side-projects the honest answer is “yes, a single modest server is correct, and adding a load balancer + cache would be over-engineering.” Knowing that is just as valuable as knowing when it’s “no.” You now decide with numbers instead of vibes, and you won’t let an AI talk you into infrastructure you don’t need.
Why is a cache so much faster than a database? Why does a CDN help a user in Sydney so much? The answer is a short ladder of speeds you can operate, and once it’s in your bones, most design feels obvious.
Latency, Throughput, and the Speed of Everything
A short ladder of numbers that secretly explains almost every design choice in the manual.
- The latency ladder, and a trick to make it stick
- Why memory beats disk and local beats global, the “why” behind caches, CDNs, replicas
- Latency vs throughput, two things beginners constantly confuse
- Why averages lie, and what “p99” really costs at scale
The idea that ties it together
Different ways of getting data aren’t a little different in speed, they differ by factors of a hundred, a thousand, a million. A designer who feels those gaps makes good decisions automatically. So let’s put the gaps in your bones. Below is the ladder of operations from fastest to slowest. Flip it to human scale, every latency multiplied by a billion, so one nanosecond becomes one second, and click a rung to feel it.
Nearly every technique in this manual is a scheme to keep work near the top of the ladder, in fast memory, on nearby machines, and avoid the bottom, distant networks and disk seeks. If cache access feels like one second, reaching another continent feels like a five-year expedition. That’s the spread you’re always designing against.
Three rules fall straight out of the ladder, and you’ll recognize them as Chapter 2’s moves: keep hot data in memory (that’s caching, promoting reads from the slow bottom to the fast top); keep users close to content (that’s the CDN, distance is one of the most expensive things on the chart); and prefer sequential over random, and batch your work (each network round trip is a trip down the ladder, make as few as you can).
A very common performance disaster: code that makes many small back-and-forth calls to assemble one response. Fifty sequential calls at ~1 ms each is 50 ms of pure waiting before any real work. The fix is almost always to batch, ask for everything at once. If you’ve ever had a page that was mysteriously slow despite “simple” logic, this was probably why. AIs generate this pattern constantly, a loop that queries the database once per item instead of once for all items.
One system I built had to pull content from many different external sources. The naive design points a single tool at every source; in practice they vary wildly, some hand you clean, structured data directly, some need a light client, some only respond to a full browser. So it became a strict ladder: try the cheapest, fastest method first, and escalate to heavier, slower tools only when a source actually demanded it. That kept resource use low while holding a high success rate.
Latency vs throughput
Latency is how long one operation takes, a single thing’s delay. Throughput is how many complete per second, volume. Picture a pipe: latency is how long one drop takes end to end; throughput is how many liters per second come out. A wide, long pipe can have high throughput and high latency, fast for the crowd, slow for the individual. Batching is the classic trade: it raises throughput (less per-item overhead) but raises latency (each item waits for the batch). A payment confirmation optimizes latency; a nightly analytics job optimizes throughput. Knowing which your feature needs prevents a whole class of mis-designs.
Averages lie: the tail
The average latency of a system is nearly useless. What matters is the slow end of the distribution, the tail, described in percentiles. p99 is the “slowest 1%” line. Why obsess over 1%? Because at scale, one page often depends on many backend calls at once, and the slowest one gates the whole page. Operate the simulator: set how many services a page fans out to, and how often each is slow.
Set N = 50 and p = 1% and watch: even though every service is “99% fast,” about 40% of page loads wait on a slow one. Now drag the slow chance down, that’s what pushing services to p99.9 buys you. Or drag N down, that’s what reducing fan-out buys you.
When a request depends on N independent services each slow with probability p, the chance the whole request is affected is 1 − (1 − p)ᴺ, and it climbs fast with N. At scale, “rare” per-service slowness becomes “common” per-request slowness. Reducing fan-out and taming the tail are core scaling skills, and the reason serious teams track p99 and p99.9, never the average.
Two architect’s instructions the ladder and the tail earn you. First, on performance reviews of generated code: “find any query inside a loop and batch it”, the single most common speed bug an AI ships. Second, when you specify observability: “instrument p99 and p99.9 latency, not averages, on every service a user-facing page depends on.” If someone shows you a dashboard proudly reading “avg 45 ms,” you now know to ask for p99, it’s routine for it to be 10× worse while a real slice of users quietly suffers. Averages hide exactly the users you most need to see.
- Latency / throughput
- Time for one operation / how many complete per second. Independent, you can trade one for the other.
- The latency ladder
- Operations ordered by speed, spanning ~100-million-fold from cache to intercontinental network.
- Sequential vs random access
- Reading a continuous stretch (fast) vs scattered spots (slow, especially on disk).
- Batching
- Grouping operations to raise throughput, usually at the cost of latency.
- Percentile (p50/p95/p99/p99.9)
- Points in the latency distribution; the tail is what users actually feel.
- Tail at scale
- The outsized impact of the slowest requests, amplified when one request depends on many services.
Using only the ladder, explain in a sentence each: (a) why a cache speeds a read-heavy app; (b) why a CDN helps a distant user; (c) why 50 sequential DB calls to build one page is bad.
(a) The cache holds hot data in memory (“2 minutes”) so most reads skip the database’s slow disk work (“months”), you promote reads up the ladder. (b) A distant server means an intercontinental round trip (“5 years”) every fetch; a CDN keeps a copy near the user, a cheap local trip high on the ladder. (c) Each call is a round trip down the ladder, and they’re sequential, so delays stack, batch them into one.
Optimize each primarily for latency or throughput: (a) search autocomplete as the user types; (b) an overnight job re-encoding your whole video library; (c) tap-to-pay checkout confirmation; (d) a pipeline ingesting a billion analytics events/day.
(a) Latency, each keystroke must feel instant. (b) Throughput, nobody waits on any one video. (c) Latency, the user stares at a spinner at the most abandonment-prone moment. (d) Throughput, no single event matters; maximize events/sec.
Using the simulator: a page fans out to 50 services, each p99 (1% slow). Roughly what fraction of loads hit a slow call? Is per-service p99 good enough here, and what are your two levers to fix it?
≈ 40%, usually not good enough for a core page. Your two levers: push services toward p99.9 or better (drag p down), or reduce fan-out (drag N down) by calling fewer services or making the page resilient to one slow laggard (render without it, fill in later). This is tail-at-scale in miniature, and why “our average is fine” is never the whole story.
You can now think in moves, in numbers, and in speeds, the three thinking tools the rest of the manual builds on. Part II examines each building block on its own, starting with the load balancer and a deep look at statelessness.
This chapter is being written
This manual is a living document. Part I is complete and fully interactive, work through it first, since every later chapter leans on its three thinking tools (moves, numbers, speeds). This chapter will be added into the manual next, in the same interactive form: live instruments, design reasoning, and a “Directing the build” panel rather than code.
The foundations are where the leverage is. Make sure the estimation console and latency ladder feel obvious before moving on.