Design Pastebin

01Problem and scope

A paste service accepts text, stores it durably, and returns a stable URL from which authorized readers can retrieve the exact saved bytes. Unlike a URL shortener, the service owns the content as well as the identifier. The design must therefore cover upload limits, publication, access checks, rendering safety and byte reclamation. For example, creating a diagnostic log returns https://paste.example/p/p7Hk2Lm9; reading that URL must return the complete saved text. Metadata describes the text—owner, length, title, visibility and expiry—rather than containing the text bytes themselves.

The scope is immutable text up to 10 MB, with public, unlisted and explicitly private access, optional custom aliases, and expiration. Clarify these visibility modes before choosing a cache: unlisted means omitted from discovery but accessible to anyone with the address; private means an authenticated grant is required. Forwarding an unlisted URL does not preserve team membership restrictions.

We support a website and programmatic API. Images, collaborative editing, full-text public discovery and executable code previews are outside this interview. Optional titles, owner listings and approximate view counts remain in scope. Request rates are modest, but retained text grows large. If metadata and bytes move to different stores, creation and deletion must remain recoverable when only one store finishes its work.

02Functional requirements

1. Create with request key: One paste identity for repeated attempts of the same payload.
2. Choose custom alias: Claim if unused; conflict rather than overwrite another paste.
3. Read public/unlisted: No login required; visibility still obeys expiry and deletion.
4. Read private: Authenticated current grant is required before bytes are delivered.
5. Delete: Owner revokes new reads; physical bytes may be reclaimed later.
6. List or inspect statistics: Owner-only cursor pages and explicitly delayed counters.

Publication and text delivery

A paste is visible only after the complete intended content exists durably. The owner may upload a large paste over a slow connection without occupying every read-serving worker. The reader receives either the exact saved text or an explicit error; the application must not render a truncated upload as a successful paste. A browser page escapes text, while the raw endpoint returns a plain-text response.

Immutability, expiry and ownership

Immutable text avoids concurrent-edit semantics. Updating a paste means creating a new version with a new identity, not mutating an already cached body. Automatic expiry is evaluated during reads, even if cleanup is delayed. Never reassign an expired alias to a stranger: old incident notes could otherwise point to unrelated text. Anonymous creation is possible with stricter quotas and a separate deletion secret; this worked design uses authenticated owners so ownership and retry identities stay clear.

03Non-functional requirements

1. Latency: Metadata lookup p95 below 100 ms; time to first byte p95 below 200 ms inside the serving region. Whole-response time is separate: 10 MB over 10 Mb/s already takes roughly eight seconds before overhead.
2. Availability: 99.9% eligible read availability.
3. Durability: Acknowledge creation only after configured replicated storage accepts it. Accepted content survives one storage-node or availability-zone failure.
4. Regional recovery: Use tested backups with an initial one-hour restore objective and an explicitly measured backup recovery point.
5. Private authorization: Check current authoritative metadata on every new request. A completed grant revocation blocks requests begun afterward; already delivered bytes cannot be recalled.
6. Public visibility: Public/unlisted reads have a 30-second bounded visibility-cache policy. Use safe server time for expiry; no cache lifetime may exceed the paste deadline.
7. Failure behavior: If a storage partition prevents current private authorization, return unavailable rather than guessing.

Publication and read invariants

Publication is the decision that a stored upload may be served. In the later two-store design, READY records that committed decision; merely uploading an object is not enough. These invariants keep the stored bytes, access policy and cleanup work in agreement.

These rules define crash behavior more precisely than the uptime percentage. The UI distinguishes uploading, ready, expired/deleted and temporarily unavailable so a service fault is not mistaken for data loss.

04Capacity estimates

Workload assumptions and arithmetic

Use these workload assumptions: one million pastes/day, five reads per paste, 10 KB average content and 10 MB maximum. 1M / 86,400 = 11.6 creations/s and 5M / 86,400 = 57.9 reads/s. A tenfold peak gives 116 creations/s and 579 reads/s. These rates fit a credible initial application; capacity pressure comes from long retention and the largest objects.

Worked estimates

Capacity implications and limits

The last row is a stress case, not the expected average. If 116 uploads/s each take five seconds, roughly 580 uploads are active concurrently. Buffering 10 MB for all of them could require 5.8 GB just for request bodies. Stream bytes with bounded buffers and admission control. For caching, 100,000 distinct hot 10 KB pastes need 1 GB of body bytes plus overhead. A percentage of read requests does not establish the number of distinct cached objects; measure unique hot IDs and byte-hit rate—the fraction of requested bytes served from cache—instead.

05APIs and contracts

Request and response example

The owner sends POST /v1/pastes with key paste-create-88 and {"text":"service-started\nrequest-42 failed","title":"Checkout log","visibility":"private","expiresAt":"2027-01-01T00:00:00Z"}. The service hashes the validated payload, including visibility and expiry, for request-identity checks. A repeated key with different content returns 409. Small requests can complete synchronously; if publication has not completed by the response budget, return 202 with the existing paste identity and status endpoint, never a false ready response.

Interface contracts

Validation and response semantics

Reject oversized content with 413, invalid text/alias with 400, exhausted quotas with 429, and temporary storage inability with 503. Private unauthorized reads should avoid disclosing whether a guessed ID exists. Custom aliases have a length and character policy; generated aliases use a larger random space and still rely on atomic uniqueness. Programmatic clients use scoped credentials and the same byte/request limits as the website, not an unrestricted “developer key” bypass.

06Data model and access patterns

This is the model used after text bytes move out of the database. The Paste row connects an object to its owner and visibility state; CreateRequest preserves the result of a retried submission; Grant answers who may read private content. The cleanup outbox is a database record of deletion work, committed with the metadata change so a crash cannot lose the obligation to remove unused bytes.

The reader first loads the paste ID and grant through an indexed query; possession of an object key does not establish permission.

Use (ownerId,createdAt,id) for owner pagination and (state,leaseUntil,id) for stale-upload reconciliation. An upload lease is the server-recorded deadline before which that upload generation is allowed to become READY. Reconciliation means checking interrupted uploads and either completing or cancelling them from their recorded state. An expiry bucket index avoids scanning billions of rows. Byte objects have immutable generation-specific names such as pastes/p7Hk2Lm9/g1/body; they are not publicly readable by possession of their storage key. Store expected size and a real checksum separately. Do not assume an object-store ETag always equals the full content hash; multipart and encryption modes can differ.

The metadata row is authoritative for whether a paste may be served. The object store is authoritative for the actual bytes. Rendered HTML, caches and counters are rebuildable. Initially all metadata transactions run in one relational database. If it is later partitioned, keep each paste, grant set and cleanup state at the same paste owner, and use a recoverable request-reservation protocol when an owner-scoped retry record cannot be colocated. Our present workload does not require that extra distribution immediately.

07Basic working design

Single-database publication

The minimum correct service has one app and one SQL database. The owner's POST validates UTF-8 and byte length, then inserts the text, metadata and request result in one transaction. The commit is the publication point. Before it, the reader cannot see the paste. After it, a lost HTTP response is handled by looking up paste-create-88 and returning p7Hk2Lm9. A conditional unique insert, rather than a preceding absence check, decides an alias race.

Authorized text reads

The reader's raw GET reads the committed row, checks visibility, grant, expiry and deletion, and streams the stored text. The HTML endpoint escapes it rather than inserting it as markup. At roughly 58 average reads/s, this implementation can be perfectly reasonable. SQL can store text; “billions of records” is not sufficient evidence to reject a relational engine without considering time horizon, partitioning and operational requirements.

Expiry, backups and scope of the baseline

08Find the baseline flaws

09Improve the design, step by step

1. Separate immutable bodies from metadata. Retained text and backup pressure trigger object storage. Reserve UPLOADING metadata, stream to a generation-specific object, verify it, then transition to READY. This keeps database queries small and body scaling independent. It costs extra requests, reconciliation and a two-store failure protocol. Keeping text in SQL remains the better alternative at small volume when one commit is more valuable than storage separation.

2. Separate transfer and read capacity. Slow uploads trigger dedicated upload pools with bounded streaming buffers; read APIs have independent concurrency limits. Larger future objects could use short-lived direct upload authorization, but the 10 MB limit does not automatically justify multipart orchestration. The improvement is read latency under slow senders. The cost is another capacity pool and partial-upload cleanup. A single event-driven pool is simpler if load testing proves adequate isolation.

3. Add public-body caching and replicated metadata. A viral log and zone failures trigger this step. Replicas protect accepted metadata; cached immutable bodies reduce origin work. Authorization still precedes private delivery, and visibility checks bound public deletion delay. Costs include RAM, eviction decisions and invalidation/freshness handling; caching everything wastes memory on one-read pastes. Retain small frequently reused bodies and separately cap very large entries.

4. Add durable cleanup and delayed counters. Expiry scans and synchronous view-count contention trigger background work. Deletion writes a cleanup outbox entry in the same metadata transaction; a relay and worker retry external deletion. Approximate view events go to a bounded queue. The improvement is predictable reads and eventual reclamation. Costs are queue retention, duplication and lag monitoring. Exact view accounting is rejected unless it becomes a billing requirement; in that case change the acknowledgement and event durability contract explicitly.

Logical metadata partitions become a subsequent step only after measured storage or write limits, not a required starting box. Hashing IDs spreads distinct pastes; replicated body caches address one hot paste. Health-aware balancing sends traffic only to ready service instances.

10Detailed architecture

Separate transfer and read pools

The final system has an authenticated upload API and an independently scaled read API behind an edge router. Both check the replicated metadata database. Upload workers stream to a private immutable object store; they cannot grant visibility merely by finishing an object write. Read workers enforce metadata policy, then retrieve bytes through an internal body cache. A public content-delivery layer can reduce geographic transfer latency under the explicit freshness contract, while private requests remain authorization-gated.

Metadata and byte-store authority

The metadata leader owns paste states, grants, request identities and cleanup intentions. Its replicas provide the chosen failure tolerance. The object store has its own replication configuration, repair and backup obligations; duplicating metadata does not protect missing text. A reconciler inspects expired upload leases and resumes or cancels them through metadata transitions. A cleanup relay consumes committed outbox rows and queues object removal. Counters are a separate derived store.

Synchronous boundary and overload control

Synchronous creation ends only when the READY transition commits; synchronous reading ends only after authorized bytes are selected and streamed. Content statistics and physical reclamation may lag. The private storage boundary is important: an internal cache hit or guessed object key cannot skip authorization. If direct download URLs are introduced later, the time during which anyone holding that URL can download without another permission check becomes an explicit revocation limitation. Any such period conflicts with the immediate private-read revocation contract; retain authorization at the delivery edge or explicitly weaken that contract before introducing such URLs.

11Write path and acknowledgement

Uploading saves the bytes; committing READY metadata permits readers to retrieve them. Request paste-create-88 reserves paste p7Hk2Lm9; each step below states what a retry or collector can safely observe.

1. Authenticate the owner, validate byte quota and text encoding, and calculate a payload identity. Reserve p7Hk2Lm9 with generation g1 and UPLOADING, together with its request key in a local transaction.
2. Stream bytes to pastes/p7Hk2Lm9/g1/body using a bounded buffer. Retry a failed transfer to the same immutable generation only with matching expected content; never reuse that identity for changed text.
3. Verify storage acknowledgement, length and checksum. This confirms the body exists but does not yet make the paste public.
4. Begin a metadata transaction, lock p7Hk2Lm9, and require state UPLOADING, matching g1, and a still-valid publication lease. Atomically set READY and save the successful request result.
5. Commit, then return 201. If the response disappears, the next request with paste-create-88 returns the committed result. If the lease expired, return a recoverable pending/failed state rather than bypassing the guard.
6. A timed-out transfer with unknown outcome is inspected by object identity and checksum. Existing correct bytes may be reused; absent or mismatched bytes are retried or rejected without exposing them.
7. If the owner abandons the operation, the reconciler eventually transitions it to GC_PENDING and reclaims g1. A late uploader cannot publish that generation after the transition.

No remote transaction spans object storage and SQL. The ordering plus metadata guards ensure a crash produces invisible recoverable work, rather than an acknowledged paste with missing content.

Make immutable object creation an enforced write rule. For an S3 implementation, a conditional create such as If-None-Match: * prevents overwriting an existing generation; on an existing-object result, verify the saved content identity before reusing it. Keep cleanup rights separate from upload rights. The product’s zone-loss promise also requires a storage class with the corresponding multi-zone durability, not a single-zone option chosen only for latency.

12Read and delivery path

Private reads must authorize against current metadata before exposing even a cached body. The example uses paste p7Hk2Lm9, reader u31 and immutable object generation g1.

1. The reader sends an authenticated GET for p7Hk2Lm9. The read API obtains the authoritative metadata and current grant decision. In this example Grant(p7Hk2Lm9,u31) exists.
2. It requires READY, an unexpired deadline, no deletion marker and allowed identity. Failure stops before body-cache lookup results are exposed. A storage outage is a 503; a denied or unavailable paste follows the chosen nondisclosure response.
3. The API requests immutable object generation g1 through the internal cache. A miss reads private object storage; simultaneous misses can share one refill. A cache entry becomes valid only after its complete length and checksum are verified. A streaming response uses storage integrity checks and may provide an end-to-end checksum; if verification fails after transmission starts, terminate the response and report failure rather than claim a complete successful body. Do not pretend a whole-body checksum can validate the first byte before the rest has arrived.
4. Raw delivery sets the plain-text type and disables content sniffing. The browser page escapes HTML metacharacters. The owner's pasted <script> remains text.
5. The API streams with backpressure: if the reader reads slowly, it does not buffer the whole object repeatedly. It enforces a per-connection and account byte budget.
6. A bounded view event is emitted asynchronously. Repeated reads can increase approximate counts; they do not mutate the authoritative paste row on every view.

A public edge copy carries an absolute visibility deadline anchored to authoritative validation, bounded by both the 30-second policy and the paste expiry. A delayed fill cannot start a new 30-second lifetime; use a conservative time margin for uncertainty. Keep browser responses no-store for the stated deletion behavior, while controlled internal/edge caches enforce those deadlines. A private body may stay cached internally longer because a fresh permission check gates each request.

13Correctness deep dive

Publication and collection share one authority

Uploader U may try to publish while collector G removes an abandoned upload. Both must check and change the same metadata row; finding an object in storage does not establish permission to publish or delete it. Require every upload to have a finite lease, and never permit an expired generation to publish without a new coordinated reservation.

Which state transition wins?

Crash recovery and retained payloads

A crash after READY but before HTTP response is recovered by the request result. A crash after GC_PENDING but before external deletion is recovered by the durable outbox. The relay may publish cleanup twice; deletion by exact immutable generation is harmless when repeated. If U continues uploading after collection, its late bytes remain invisible and a subsequent sweep removes them. Bound upload credentials and maximum transfer duration to prevent indefinite orphan recreation. The proof depends on enforced state guards, not on hoping a grace interval exceeds every slow request.

14Failure and recovery

Backup and recovery boundaries

Replicas protect node/zone failures, while backups protect accidental deletion and corruption according to tested recovery points. Restore metadata and body generations together and check every sampled READY reference. A residual regional-disaster loss window remains unless synchronous regional durability is added; naming an object-store product does not eliminate that tradeoff.

15Operations, security, and cost

Integrity, queue and latency signals

Alert on any READY row whose referenced object is absent, the oldest UPLOADING lease, orphan bytes, cleanup-outbox lag and expired content still served. Separate latency for metadata, first byte and whole body. Track cache byte-hit ratio as well as request-hit ratio: a cache serving many tiny pastes may still leave large origin egress. Count authorization denials and request bytes independently to detect enumeration and upload abuse.

Metadata and body-cache cost

For one million daily creations, storing a 200-byte cleanup/request bookkeeping row per paste adds about 200 MB/day before indexes. Ten years would add roughly 730 GB of such raw metadata if never compacted. Define retention for completed request keys and compact permanent alias claims rather than accidentally keeping every transient log forever. The dominant cost remains retained bytes and copies, followed by object operations and delivery. Compression may reduce text bytes, but enforce decompressed size limits and checksum the canonical intended representation.

Dual-read storage migration

Roll out object storage behind a dual-read migration: write new pastes to the new scheme, copy older bodies, verify lengths/checksums, switch a row's pointer transactionally, then remove the old database payload after a rollback window. Do not delete the old copy merely because a copy job started. Test crash points, expired upload credentials, a collector/uploader interleaving, denied private-cache hits and backup restoration.

Safe rendering and privacy

Authentication, rate limits and safe rendering are required before public launch. Avoid recording secret paste contents in request logs or analytics. Deduplication, if added, must not reveal whether another tenant owns matching text; shared object reclamation then requires transactional references or a verified reachability process before deletion.

Restoring permanent alias claims

Restore permanent alias claims as well as visible pastes. If a regional recovery point loses some recent claims, freeze new allocations in the old namespace until claims are recovered; a fresh namespace can host new random pastes without giving an old incident URL a new owner. Also do not restore old private grants as current authority without reconciling revocations that may have occurred after the backup.

16Decision ledger and limitations

Keeping text in SQL and moving it to object storage are both valid choices. The deciding issue is whether lower byte-storage and backup pressure justify coordinating publication across two stores. The table makes the consequences of that split explicit alongside the access and counting policies.

Six characters from a 64-symbol alphabet offer about 68.7 billion combinations, but a finite namespace does not make random draws unique or make guessing impossible. We choose a larger generated namespace, enforce uniqueness at insertion and rely on identity-based grants for private content. A preallocated key-generation service could reserve batches durably; it adds failover coordination and wastes uncertain unused batches on crashes. At 116 peak creates/s it is unnecessary.

We do not promise that revocation deletes copies the reader already saved, or that a public paste remains confidential because its URL is obscure. We also do not claim a SQL database must be replaced by a key-value store at a specific row count. The justified boundary is small indexed metadata versus independently retained immutable bytes.

17Interview closing

“I am building an immutable text-sharing service: the owner publishes one paste and the reader receives its exact authorized bytes. The workload is around twelve creates and fifty-eight reads per second on average, but ten-year content retention reaches 36.5 TB before copies. I would start with text and metadata in one transaction, then separate bodies when storage and backup pressure justify the complexity.

“The final design reserves an upload generation, writes and verifies the object, and only then commits READY metadata. Private reads authorize before delivery. A metadata state transition arbitrates publication against cleanup, so a collector cannot remove bytes that a late uploader is still allowed to publish. Lost responses recover through the same request identity. Public caching and delayed statistics improve read cost; they do not become the authority for existence or permission.

“I accept temporary unavailability when current private authorization cannot be established. My next measurements are how many large-paste transfers overlap during peak traffic, whether every READY record points to complete content, and how long unused upload objects wait for deletion.”

If the interviewer adds editable pastes, preserve immutable body versions and atomically switch a metadata version pointer with an expected-version precondition. Explain whether readers get the latest version or a stable historical link. Do not overwrite the old object under a cacheable key and assume every cache changes simultaneously.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.