Design a distributed object store

01Problem and scope

An object store maps a tenant-scoped key to one complete immutable version. Keep the metadata mapping a key to a version separate from the storage locations holding its bytes. Parts upload independently, but the key points to the new version only after the database commits its verified manifest: the ordered record of chunks, lengths and integrity information. This design provides resumable multipart upload and strong per-key visibility within one region rather than shared-file mutation. A two-GiB object at T7/report.pdf uses thirty-two 64-MiB parts; completion U31 competes against expected version V4.

Smallest working design

Start with one server: write bytes to a temporary location, verify them, then atomically update the key’s metadata to point at the completed file. The key is a namespace entry; the underlying bytes can be immutable. Distributed object storage extends placement and recovery while preserving that publication idea.

Clarify the object contract

Candidate: “Must readers see a partial overwrite, and can uploads resume?” Interviewer: “Never partial; support multipart resume.” Candidate: “Do we need shared-file mutation or only whole-object versions?” Interviewer: “Whole objects, with strong per-key visibility in one region.” This makes immutable byte chunks plus an atomic namespace pointer a natural starting design.

Protocol cases to prove

The protocol must handle interrupted part uploads, lost acknowledgements and two uploads both trying to replace the same expected version. Each read must retain one manifest throughout the response, called pinning that version, so an overwrite cannot change the remaining chunks mid-read. The hard promise is not that all chunks travel atomically over the network. It is that the key names one complete verified version only after the publication transaction commits.

02Functional requirements

1. Operate on private objects. Support PUT, GET, HEAD and DELETE with metadata/checksums; private access is the default.
2. Upload in parts. Initiate, upload/list/retry parts, complete or abort, and recover status after a lost response.
3. Publish complete versions. Completion supplies an ordered part set with integrity metadata; missing parts fail without changing the current object.
4. Read one version. GET/HEAD resolve the current version or an authorized explicit version. Range reads return the requested inclusive byte range and correct metadata; invalid ranges fail clearly.
5. List with bounded cursors. Use an opaque tenant/prefix/last-key cursor over lexicographic namespace keys.
6. Retain or delete versions. Support optional version retention. Deleting the current key follows that policy and does not automatically erase every historical byte.
7. Delegate bounded access. Signed grants can authorize a key/version/upload operation under bounded expiry and headers.
8. Control lifecycle and quotas. Bound object size, active uploads and retained bytes; abort abandoned sessions, expire permitted versions and schedule repair/cold-tier moves without exposing incomplete manifests.

Write and retry contract

Choose strong per-key visibility in one region: after a committed overwrite, new current-version reads resolve the new complete version. A request may pin an older explicit version where policy permits. A retry with the same upload/part identity and checksum confirms the same logical work; replacement parts, if allowed before finalization, need an explicit generation. Multipart sessions cannot become unlimited unbilled temporary storage.

Listing is not a snapshot

Each page reads currently committed namespace rows. A whole traversal is not a point-in-time snapshot: concurrent insertions before the cursor may be absent, and later keys can change between pages. An inventory/export needing a fixed view uses a separately retained namespace snapshot.

Scope and provider boundary

In-place byte edits, filesystem locking and multi-key transactions are outside scope. Amazon S3 documents strong read-after-write for PUT/DELETE and offers versioning; these are provider-specific capabilities, not universal object-store axioms. Cross-region replicas, external metadata databases and CDNs may have separate freshness contracts. S3 overview/consistency.

03Non-functional requirements

1. Workload assumption. Ten million new objects/day averaging 10 MB; one billion reads/day averaging 4 MB; thirty-day baseline retention.
2. Latency. Metadata operations p95 below 100 ms and first-byte reads p95 below 200 ms for healthy hot data within the region. Byte count and network rate dominate whole-transfer time.
3. Availability. Target 99.95% API availability under the assumed deployment.
4. Durability before publication. For frequently accessed, or hot, writes, require three durable full copies across independent failure domains: groups of storage resources placed so that a single stated failure does not destroy every copy. Metadata commits have their own replicated durability policy.
5. Access isolation. Authorize before resolving private content or issuing a grant. Version-keyed immutable caches must enforce the access contract.
6. Bounded lifecycle. Bound incomplete-upload lifetime and protect active readers, uploads and retained versions during garbage collection.

Per-key consistency invariants

Limits of these promises

The durability promise covers the stated single-node/domain failure; it is not a fabricated universal “eleven nines” claim. Repair restores redundancy. Correlated failures, operator errors and regional loss require further design and testing.

A CDN cached under an unversioned name can have a different freshness policy; do not silently include it in the strong-origin guarantee.

04Capacity estimates

Assume 10M new objects/day averaging 10 MB, thirty-day retention, and 1B daily reads averaging 4 MB.

Interpret byte throughput

These are hypothetical sizes. Request rate alone would badly understate the network problem. Cache popular immutable versions and support ranges where clients need only part of a file; account for egress and repair bandwidth separately.

Read and upload throughput

At a fivefold peak, read payload can approach 231.5 GB/s and ingress 5.8 GB/s under these assumptions. Large objects make bandwidth and disk throughput more important than the modest 116 new-object requests/s average. Multipart upload also increases request count: the uploading client's 2 GiB object produces 32 part writes plus initiate/complete operations, not one write call.

Metadata overhead

Thirty days yields roughly 300M objects. At an illustrative 1 KB namespace/manifest header each, metadata is about 300 GB before chunk references, indexes and replicas. A 2 GiB version with 32 chunk references at 64 bytes adds about 2 KB of reference metadata. Small objects may have disproportionate metadata cost and should avoid unnecessary multipart/chunk overhead.

Coding and repair bandwidth

An erasure code stores data fragments plus calculated parity fragments that can reconstruct missing data. A 4+2 code stores four data and two parity fragments, totaling 1.5 times the original bytes. It reduces 3 PB raw from 9 PB at three replicas to about 4.5 PB encoded payload, saving 4.5 PB before extra capacity reserved for placement and repairs. It spends CPU/network during encoding and repair. Reconstructing a missing fragment may read several surviving fragments; budget repair traffic separately so a disk failure does not starve the 46.3 GB/s ordinary read stream. Popular immutable versions can use caches, but cache hit ratio determines actual origin savings.

05APIs and contracts

POST /uploads
{key:"T7/report.pdf",size:2147483648,expectedVersion:"V4"}
→ {uploadId:U31,partSize:67108864,state:"open",expiresAt:...}

PUT /uploads/U31/parts/9
Content-Length:67108864; checksum:<defined algorithm/value>
→ {part:9,generation:1,checksum:...,durabilityStatus:"stored"}

POST /uploads/U31/complete
{parts:[{number:1,generation:1,checksum:...},...32]}
→ {version:V5,state:"completed",length:2147483648}

Completion validation and replay

Completion rejects missing parts, mismatched lengths/checksums, expired/aborted sessions and changed expected key version. The service encodes the ordered completion manifest in a documented, consistent format and records its hash, called the completion fingerprint; a retry with another part set cannot be mistaken for the same request. GET /uploads/U31 recovers the committed V5 outcome after a lost reply. API examples describe this designed service, not literal S3 request syntax.

Version reads and listing

GET/HEAD accepts an explicit version or resolves current; conditional writes use expectedVersion rather than comparing client wall clocks. DELETE can accept an expected version to avoid accidentally deleting a newer overwrite. Listing cursors bind tenant/prefix and chosen namespace view, with bounded expiry. Authorization derives tenant ownership from verified credentials; the string T7/ is not proof that the uploading client may access that key.

Checksum and ETag semantics

A checksum states both its algorithm and which bytes it covers, such as one part or the whole object. An entity tag (ETag) is a provider-defined version validator and not universally the full-object MD5, especially for multipart objects.

06Data model and access patterns

The identities describe different layers: a part identifies a position in one upload, a chunk identifies immutable stored bytes, and a version manifest assembles chunk references into a complete object. The logical key points to one current version. Keeping these layers separate allows part retries and byte-placement repairs without silently changing a published object.

Part identity and replay

At 64 MiB per part, 2 GiB / 64 MiB = 32 parts. Part numbers and upload identity make retries address the same work. An object manifest is the committed map from logical byte ranges to stored chunks. Metadata stores upload state, expected prior version, quotas, retention, and ownership; storage nodes do not decide application authorization.

Upload and chunk records

Add Upload(U31,tenant, key, state, expectedVersion, completionFingerprint, expiry, resultVersion), Part(U31,number, generation, checksum, chunkRef) and KeyHead(T7,key, currentVersion). Version manifests are immutable after publication and include ordered ranges, ownership, retention and integrity data. Chunk placement metadata tracks replica locations, checksums, health and placement generation.

Namespace partitioning

Partition namespace metadata by tenant/key hash or range according to listing needs. All conditional publication state for one key/upload must share a transactional authority or use a specified atomic protocol. Byte nodes own chunk storage/verification; they do not choose which version a logical key names. A replicated metadata leader handles that decision.

Collection roots

Garbage collection (GC) deletes chunks that are no longer needed. A root is a recorded reason to retain bytes, such as a current or retained object version or an active upload. A reader pin records that a download still needs its chunks. Track Chunk(chunkId, state=LIVE|DELETING|DELETED, uploadRefs, versionRefs, readerPins, deletionGeneration) under the same metadata authority that owns the upload and version references. This design does not deduplicate chunks across independent authorities. Create the protected upload reference before granting a byte upload. Every new upload reference, retained-version reference or reader pin is acquired atomically only while the chunk is LIVE; DELETING is irreversible for that chunk identity. Namespace listing reads metadata, not a scan across disk directories. Placement can change during repair while a manifest keeps the same logical chunk identity.

Integrity is not authorization

Keep integrity and access separate: a correct checksum proves bytes match expected content, not that the caller is entitled to receive them. Signed grants and metadata ownership enforce access before storage-node requests are issued.

Enforce immutable bytes

A chunk ID is immutable because the byte service enforces it, not because the name looks unique. Its first accepted write atomically creates the bytes and their checksum; a retry may confirm identical bytes but cannot overwrite that identity. Replacing a multipart part allocates a new chunk ID and part generation. The upload grant binds upload, part generation, chunk ID, allowed size/checksum and expiry; the node checks those restrictions. If managed storage backs the byte plane, retain an exact immutable VersionId or enforce create-only conditional writes and deny bypass writes. A reusable signed PUT to a mutable key does not implement this invariant.

Current head versus historical roots

A current KeyHead is always a version root even when optional historical retention is disabled. Replacing/deleting the head removes that current root under the same authority, but historical retention roots and reader pins may keep the old version alive. This distinction prevents a non-versioned bucket from collecting its still-current object.

07Basic working design

Temporary bytes and metadata

The baseline has an API, a transactional metadata database and one local disk. The uploading client initiates U31 against V4. Each part is written to an immutable temporary file, fsynced under the stated local durability policy and recorded by part identity/checksum. A lost part reply can be retried and compared with the recorded part rather than restarting two GiB.

Freeze, publish and pin reads

1. Freeze and verify. Completion first freezes U31's chosen part generations, validates the ordered set and builds a manifest.
2. Commit current version and retry result. It then atomically updates KeyHead from V4 to V5 and records U31 completed/result V5.
3. Read one immutable manifest. Readers consult KeyHead once, load that immutable manifest and stream its chunks.
4. Before/after visibility. Before the transaction they see V4; after it they see V5.
5. Never publish through a part rename. No rename of an individual part makes partial V5 visible.

Crash recovery and local durability

If the process crashes before publication, U31 can resume/finalize or eventually abort; V4 remains current. If it crashes after commit before replying, the retry reads U31's recorded result. The baseline demonstrates complete-version publication and safe retries, but its crash recovery depends on the local disk and commit policy actually implemented. It cannot survive losing its sole disk, and the disk/network become clear capacity bottlenecks.

Bytes before namespace publication

The same ordering will apply when storage is distributed: save durable bytes first, then publish metadata that names the complete object.

08Find the baseline flaws

09Improve the design, step by step

Replication stores full copies; three replicas cost roughly 3× raw bytes and can tolerate selected node/domain failures according to placement. Erasure coding splits data into k data fragments plus m parity fragments; an illustrative 4+2 code uses 6/4=1.5× raw payload and can reconstruct from any four valid fragments under that code’s assumptions.

Separate fragment locations across the actual failure domains promised. Six fragments on one failing disk do not survive a disk loss. Acknowledgement must specify which durable placements exist before success.

1. Distribute immutable chunks with replicated hot writes

• Trigger: disk capacity/throughput and single-node loss.
• Mechanism: Placement selects three independent failure domains; publication waits for the policy's verified durable copies. This adds parallel byte capacity and survives the promised failure.
• Benefit, cost and alternative: Costs are roughly 3× stored bytes and write network; correlated placements defeat the benefit. One disk remains suitable only for a weaker local-development contract.

2. Shard and replicate metadata authority

• Trigger: namespace growth and a metadata single point of failure.
• Mechanism: Route each key to a leader-backed shard with atomic expected-version publication and durable upload results.
• Benefit, cost and alternative: This scales independent keys, but introduces routing, rebalance and safe failover. For any one key, one metadata authority still determines the order of conditional publications. A large single metadata database is simpler until measured limits justify sharding.

3. Ranges and immutable-version caching

• Trigger: 46.3 GB/s read payload.
• Mechanism: Versioned caches and range requests avoid repeated or unnecessary origin bytes.
• Benefit, cost and alternative: Costs are cache storage, access-aware keys and stale unversioned-name risks. A download needing an entire infrequently accessed object gains little from many tiny range requests; combine adjacent storage reads to avoid unnecessary request and disk overhead.

4. Background coding, scrubbing and lifecycle GC

• Trigger: three-copy byte cost and latent corruption.
• Mechanism: Infrequently accessed versions can move to verified erasure-coded storage before old replicas are retired. Scrubbers periodically read stored bytes and check their checksums, allowing repair before corruption destroys the last usable copy.
• Benefit, cost and alternative: This saves storage while adding repair CPU/network and transition state. Keep hot small objects replicated when latency and repair simplicity outweigh byte savings. GC retains chunks referenced by committed versions, active uploads or readers until those references can be safely released.

10Detailed architecture

Namespace authority and placement

Clients enter an authenticated metadata/API gateway for namespace and upload operations. A key router resolves the metadata shard. Its leader and replicas own KeyHead, upload state, immutable manifests, quotas and publication results. A placement service maps chunks to storage nodes across failure domains; it can change healthy locations without changing an object's logical version.

Upload and background repair

For uploads, the API can return limited part grants so clients send large bytes directly through the byte path. Storage nodes verify length/checksum and report durable placement evidence to the upload metadata path. A completion coordinator freezes the manifest, validates policy, then performs the key/version transaction. There is no implication that metadata replicas contain the object bytes.

Version-pinned range reads

For reads, the gateway authorizes and resolves one version, then a range/data service obtains the relevant chunks from a versioned cache or healthy placement. Repair/scrub workers operate asynchronously and update placement state only after verified replacement data exists. Lifecycle GC scans metadata roots and staged uploads before deleting unreferenced chunks.

Why background actors matter

The final graph includes these background actors because byte durability depends on continuous repair, not just initial copying. Cross-region copies and CDN behavior are additional boundaries with their own freshness/RPO policies; the regional KeyHead guarantee cannot be casually extended to them.

11Write path and acknowledgement

Before transferring a part, the service records an upload reference that prevents cleanup from deleting its chunk. Completion freezes and verifies the manifest, then changes the key pointer and upload result atomically.

Numbered upload and publication trace

1. Authorize and create upload state. The API authenticates the uploading client for tenant T7, checks quota, and records U31 pending against current V4.
2. Choose independent placements. The placement service assigns part/chunk destinations across chosen failure domains.
3. Upload and retry immutable parts. The uploading client sends 32 parts. Part 9’s response is lost; retrying U31/part 9 with the same expected checksum confirms that immutable part generation; different bytes require a new generation, not an overwrite of verified bytes.
4. Verify the complete set. Complete verifies the ordered part set, total length, and supported integrity checks; missing/corrupt parts prevent publication.
5. Publish conditionally. The metadata transaction conditionally changes the key from V4 to complete V5 and records U31 completed. Readers saw V4 until this commit.
6. Recover the recorded result. If the completion reply disappears, querying/retrying U31 returns the recorded V5 outcome rather than publishing another inconsistent version.

Provider-specific multipart option

S3’s documented multipart initiate/upload/complete model is one implementation option. Multipart upload.

Guard each part record

The metadata write checks that U31 is still open and records the exact part generation. The completion request atomically changes open → finalizing and stores its canonical ordered manifest fingerprint. New/replacement part commits are rejected after that transition; in-flight byte uploads may finish as unreferenced objects but cannot alter the frozen manifest.

Verify the frozen manifest

The completion worker verifies total length, each required immutable part generation/checksum and sufficient durable placements. If a storage failure reduced the policy below its publication threshold, repair or request reupload before continuing. Validation failure leaves current V4 unchanged and reports a recoverable or terminal upload state according to the error.

Atomic publication result

The final metadata transaction checks U31 is finalizing with that fingerprint and KeyHead still equals V4, then inserts immutable V5, updates the head and records U31 completed/result V5 atomically. A competing overwrite that already changed the head makes this transaction fail cleanly; its bytes are retained briefly for an explicit retry/rebase policy or GC, not silently published over the winner.

12Read and delivery path

Check access and hold a reference to one immutable version before reading ranges. Cleanup keeps its chunks while a retained version or active reader still needs them.

Resolve one manifest

A reader resolves V5 once, then maps its requested range through the manifest. The half-open interval [128 MiB, 192 MiB) is the third 64-MiB part; the corresponding inclusive HTTP range is bytes=134217728-201326591. Unrelated parts need not be fetched. Pinning the version prevents a concurrent overwrite from mixing chunks from V4 and V5 in one response.

Verify bytes

Numbered range-read flow

1. Authorize and resolve one version. Authenticate the requester and validate ownership or the signed grant, including method, key/version and effective expiry. Resolve current KeyHead once if no explicit version is supplied.
2. Atomically pin live chunks. In the owning metadata transaction, resolve/load V5 and acquire reader pins for its required chunks only if their state is LIVE. A retained version root already protects its manifest. If a chunk is DELETING, do not stream from a remembered location: fail an explicitly deleted-version read or re-resolve/retry a current-key read. Validate range bounds against byteLength and choose the covered chunks/offsets.
3. Use a versioned access-aware cache. Check a cache keyed by tenant/access boundary, V5 and chunk/range identity. Never substitute an unversioned cached V4 body merely because the key name matches.
4. Read and verify required bytes. Resolve healthy chunk placements, read required bytes and verify integrity under the supported checksum/range scheme. A whole-part checksum may require verifying a larger chunk than the requested subrange unless subchunk checksums exist.
5. Recover from valid redundancy. On corruption/unavailability, try independent valid redundancy, schedule repair and fail clearly if the promised data cannot be reconstructed. Return correct content-range/length/type metadata and stream one version throughout.

Concurrent overwrite behavior

An overwrite committed during step four affects later new reads, not this pinned response. A delete may prevent new current-key resolution while an already-authorized, protected version read finishes under the defined policy.

Release pins and choose ranges

Release the durable reader pins only after the streaming service finishes its use of the chunks. A crashed streaming worker can leave reader pins behind. Recovery removes them only after preventing that worker generation from issuing further reads and waiting for its outstanding reads to finish or terminate; elapsed time alone does not prove that the chunks are unused. This deliberately favors temporary retained bytes over deleting a chunk still in use.

13Correctness deep dive

Freeze before namespace publication

First freeze the upload’s exact part list so it cannot change during validation. Then publish the key’s new version with an atomic update that orders competing uploads. These are separate state changes.

transaction freeze(U31, requestedParts):
  lock upload U31
  require canonical(requestedParts) matches any recorded completion fingerprint
  if COMPLETED: return recorded result
  require OPEN and not expired, or matching existing FINALIZING fingerprint
  # expiry applies to starting finalization; finalizing recovery uses its stored manifest
  store immutable part-generation list and fingerprint
  set state=FINALIZING; commit

verify frozen lengths, checksums and durable placements

transaction publish(U31, V5):
  lock upload U31; lock KeyHead(T7,report.pdf)
  if U31.COMPLETED: return U31.resultVersion
  require U31.FINALIZING and verified fingerprint matches
  require KeyHead.version == U31.expectedVersion  # V4
  lock frozen chunk reference rows; require each state == LIVE
  insert immutable manifest V5 and its retained-version references
  transfer U31 upload references to V5 references atomically
  set KeyHead=V5; set U31=COMPLETED,resultVersion=V5
  commit; return V5

Guard part metadata

Part updates and freezing use the same upload-state guard. Once the upload is FINALIZING, a new part update sees that state and is rejected before commit. The verified manifest names immutable chunks, so a later write to the same pathname cannot change its bytes. Placement health may change after verification; the redundancy policy is designed to survive the stated failure, while repair maintains it. Do not claim protection against arbitrary simultaneous loss between two instructions.

Competing completions

U31 wins: it locks KeyHead at V4, commits V5 and its completion result. U32 expecting V4 then sees V5 and conflicts. U32 wins: U31 fails the same predicate, leaving V6 current; it cannot publish V5 merely because its upload finished first. U31 reply lost: retry finds COMPLETED/V5 and returns it without another pointer change.

Crash and replay result

Garbage collection shares authority

Garbage collection must use the same metadata transactions as publication and reader-pin creation. If it merely checks for zero references and deletes later, a new reader could acquire a reference between those two steps:

Atomic deletion claim

transaction claimForDeletion(chunk):
  lock chunk and associated reference state
  require chunk.state == LIVE
  require uploadRefs == 0 and versionRefs == 0 and readerPins == 0
  require all abandoned owning uploads are durably ABORTED
  set state = DELETING; increment deletionGeneration
  commit deletion work(chunkId, deletionGeneration)

Abort versus publish

An abort and publication contend on the upload state: ABORTED prevents publication, while COMPLETED transfers protection to the retained version. GC cannot reclaim an OPEN or FINALIZING upload by just observing an old timestamp. Physical workers delete only the immutable identity claimed by that DELETING generation and retry until removal is recorded. No new upload, repair publication, version reference or read pin can resurrect that identity; a later upload uses a new chunk ID.

Reference versus deletion outcomes

14Failure and recovery

Observe repair and recovery

Measure byte throughput, first-byte/range latency, checksum failures, incomplete-upload age, repair backlog, replication lag, metadata conflicts, and storage overhead. Test interrupted parts, lost completion replies, corrupt replicas, concurrent overwrite, stale CDN content, and expired signed URLs. Report each boundary’s guarantees rather than saying all storage is simply consistent.

15Operations, security, and cost

Scoped grants and access enforcement

An authorized service can issue a short-lived signed upload/download URL tied to the operation, key/version, and permitted headers. Anyone possessing it may exercise that capability until its effective expiry; it is not inherently single-use. S3’s presigned URL documentation also explains credential-lifetime effects. Presigned URLs. Avoid logging grants, scope the signer narrowly, and do not expose storage credentials to clients.

Retention and abuse limits

Version retention helps recover overwrites but costs bytes and complicates deletion policy. A delete marker can hide the current name while older versions remain retrievable to authorized callers. Garbage collection removes only unreferenced expired versions/parts after respecting active uploads, retention, and recovery policy; abort abandoned uploads explicitly.

Service and repair metrics

Measure successful-byte throughput, first-byte p95, range read amplification (storage bytes fetched divided by bytes requested by the client), checksum failures, desired-versus-actual replica count, repair age, metadata conflicts and abandoned-upload bytes. A healthy PUT success rate can hide a growing repair backlog that reduces failure tolerance. Track bytes by tenant and lifecycle state so incomplete uploads cannot quietly consume unlimited capacity.

Stored bytes and recovery cost

The 3 PB retained workload costs 9 PB under three replicas versus about 4.5 PB for illustrative 4+2 coding before overhead. The comparison excludes encoding and repair bandwidth; benchmark reads and one-domain recovery before moving cold data. If a popular immutable version has 90% cache hit rate, its origin read bytes fall tenfold, but cache egress and authorization still cost resources.

Compatible rollout and fault drills

Roll out a new manifest or checksum format with readers that understand both before enabling writers. Test interrupted parts, completion after concurrent overwrite, corruption with one replica unavailable, lost completion replies and GC while a reader is pinned. For cold-tier conversion, publish a new verified placement only after every needed fragment meets policy, then retire old replicas gradually. Never use a storage-node filename or ETag as a universal authorization or full-object-integrity proof.

Reusable grants and immutable versions

Expiry normally governs admitting a request; an already admitted transfer can continue under the service's stated policy. A promise to terminate bytes immediately on revocation requires a serving-layer cancellation protocol and cannot be inferred from a presigned URL.

16Decision ledger and limitations

Storage and publication choices

Cold-data and locality limits

Replication protects against selected live failures; backups/version retention protect different mistakes. Erasure coding reduces bytes but adds reconstruction work, and its failure tolerance depends on independent placement of fragments. Cross-region replication introduces latency, cost and possibly a nonzero recovery-point gap. For any chosen policy, state which failures an acknowledged object survives.

What remains outside this design

Strong per-key origin visibility does not promise atomic transactions across multiple objects, immediate CDN invalidation or a consistent snapshot of a huge listing unless separately implemented. Signed URLs delegate a bounded capability and can often be reused until effective expiry. Their convenience does not make them single-use or instantly revocable without additional enforcement.

17Interview closing

Rehearse the architecture and contract

“I separate the logical object name from immutable bytes. Multipart uploads make parts independently retriable under a stable upload identity. Completion freezes the exact part generations, validates integrity and durable placement, then atomically replaces the expected key version with a complete new version and records the upload result. Two uploads expecting the same prior version cannot both win, and a lost completion reply returns the recorded version. Reads resolve one manifest and pin it, so an overwrite cannot mix chunks from different versions.

Defend the critical boundary

“I scale the byte plane across failure domains, keep metadata in replicated key authorities and use ranges/versioned caches for read bandwidth. Hot replicas simplify low-latency reads; cold erasure coding saves storage with repair cost. Garbage collection atomically marks chunks DELETING only after retained-version, upload and read references are absent. Reference acquisition and publication use the same metadata guard, so a new reader or publisher cannot race a deletion check. The main bottleneck here is tens of GB/s of reads and petabytes of retained bytes, not just request QPS. My next tests are completion races, corrupt-chunk recovery and one-domain repair under live load.”

Answer the follow-up

If the interviewer asks for a shared mutable filesystem, explain that byte-range mutation, locks and namespace semantics need another contract. If they ask for globally immediate reads after a regional write, revisit replication/coordination and latency rather than assuming the single-region KeyHead guarantee extends across asynchronous replicas and caches.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.