Capacity estimation: throughput, latency, concurrency and storage

01Capacity estimation and its units

Capacity estimation translates a workload into the resources needed to meet its targets. A workload specifies what users do, how often, how large their requests are and how concentrated the traffic becomes. The estimate should be accurate enough to choose a design; a load test must later measure the actual implementation. Begin with four separate ideas. Throughput is completed work per unit time, such as 500 uploads per second. Latency is how long one operation takes. Concurrency is the number of operations in progress at once. Storage is how much retained data exists at a point in time.

A restaurant can serve many meals an hour while one customer's meal takes a long time. A batch service can likewise have high throughput and high latency. More concurrent work helps use idle resources; once the limiting resource is fully busy, extra work mostly waits. A claim of “10,000 users” needs to say what they do and when.

QPS means queries per second; in an API discussion people often use it for requests per second, so state whether you are counting API requests or database queries. Network capacity, often called bandwidth, is the maximum data rate a link or path can carry under stated conditions, usually measured in bits/s. Network throughput is the rate actually achieved. Requests/s × bytes/request estimates the required transfer rate; provision capacity above that demand, including protocol overhead and headroom. Peak means the busiest declared interval, while an average spreads all work over the entire measured period. These are different quantities even when one calculation produces another.

02Worked example: average and peak QPS

Define a workload before estimating resources. For this photo-service example, assume one million daily active users, 20 photo views and 0.1 uploads per user per day, a 2 MB average upload, a 100 KB thumbnail, and 1 KB of metadata per photo. Use decimal units: 1 KB = 1,000 bytes, 1 MB = 1,000,000 bytes, and one day = 86,400 seconds.

The calculation is a sequence: first count actions per day, then divide by seconds per day, then apply an explicitly assumed peak factor. For these inputs, 1,000,000 × 20 = 20,000,000 image views/day, 20,000,000 ÷ 86,400 ≈ 231.5 views/s average, and 231.5 × 10 ≈ 2,315 views/s peak. We size thumbnail delivery against the last rate, then verify it against measured bursts.

03Storage, retention and network bandwidth

The photo workload creates two different demands: storage for retained objects and network capacity for repeated delivery. Logical data counts one copy of each retained object; replicas and backups consume additional physical storage. Keep those counts separate from bytes sent to viewers:

Keep these quantities separate:

1. Count other stored data separately. Thumbnail variants and backups are additional categories; the replication multiplier does not include them.
2. Bytes and metadata scale differently. Their large size difference is a reason to store them separately. The metadata database need not carry every byte transferred to viewers.
3. Convert units explicitly. Network links are often rated in bits/s: multiply bytes by eight. State whether you mean MB or MiB.

04Little’s law and latency percentiles

Request rate alone does not tell us how many connections or request buffers are occupied. A request continues using some resources while it waits for storage or another service. To size those resources, relate the completion rate to the time each request remains in the service.

If average time rises to 0.5 seconds while admitted traffic stays at 2,000/s, concurrency becomes about 1,000. The extra 900 requests need memory, sockets, and possibly database connections. An unbounded queue hides overload briefly while increasing latency. It does not create processing capacity.

The stable-system condition matters. If arrivals stay at 1,200/s while only 1,000/s complete, an unbounded backlog grows by 200 requests/s, or 12,000 requests in one minute. There is no steady finite average latency to insert into this calculation. Bound the queue and reduce admissions, or increase the bottleneck’s measured service capacity.

05Bottlenecks and failure headroom

A bottleneck is the resource that first limits the workload: for example, CPU, database writes or network transfer. Headroom is spare capacity reserved for bursts, uneven load and failures. Once a load test identifies the limiting resource, size enough instances to meet the target even with the chosen failures.

Assume a load test measures 800 requests/s per application instance while meeting the latency objective, and the target peak is 2,315 requests/s.

The notation ceil(x) means the smallest whole number at least as large as x; a partial server cannot satisfy the remaining load. For a chosen maximum of 70% of tested capacity after one failure:

1. Budget each survivor: 800 × 0.7 = 560 requests/s.
2. Find the survivors needed: ceil(2,315 / 560) = 5.
3. Add failure capacity: five survivors require six instances.

This is illustrative sizing, not a universal 70% rule. Real benchmarks, cost, autoscaling lag and failure domains determine the target.

Check downstream amplification

The database, network, and object store must support the same workload. Six application servers do not help if they all wait for one slow query. Estimate the read/write amplification: if each API call issues five database queries, 2,315 API calls/s can become 11,575 database queries/s.

Size CPU from CPU time

Compute demand has a different unit from elapsed latency. If a measured request uses 2 ms of CPU time:

1. CPU demand: 2,315/s × 0.002 CPU-seconds = 4.63 CPU-seconds/s, about 4.63 fully busy cores.
2. Utilization headroom: at a chosen 70% limit, ceil(4.63 / 0.7) = 7 usable cores before additional failure capacity.

06Cache working set and cost model

A cache stores copies of reused data. Its size depends on distinct hot entries, not total requests. Suppose 500,000 frequently viewed photo records occupy 1.4 KB each including key and bookkeeping overhead. That is about 700 MB per full cache copy. Ten million reads of those same entries do not require ten million stored entries.

A cache hit finds the requested value in the cache; a miss must fetch it from the underlying database or storage service, called the origin. The request hit rate is the fraction of cacheable requests served as hits. This rate turns the delivery estimate into an estimate of work still reaching the origin.

A 95% request hit rate reduces 2,315 cacheable lookups/s to about 2,315 × 0.05 = 116 misses/s under the same workload. But when the cache is empty, the origin can suddenly see all 2,315/s. Protect that origin and warm popular entries gradually. Track byte hit rate separately: a few missed large images may dominate bandwidth despite a high request hit rate.

For cost, write a symbolic model before using current provider prices: storage GB-month + read/write operations + delivered GB + compute time + replication/backup. A cheaper storage tier can have retrieval fees and slower access. The interview value is identifying the dominant cost and a way to measure it, not memorizing a vendor price that may change.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.