Design public post search

01Choose public keyword search and its limits

Design keyword search over public short posts. A query can combine terms with explicit AND or OR and request newest-first results. Add lexical relevance as a discussed extension; most-liked and personalized ordering require their own feature and candidate-selection contracts. The source posting service already exists and remains responsible for accepting posts and determining current visibility.

Use three running documents:

The query solar AND battery should match T101 and T103, not T102. This small example lets the interviewer verify the search semantics before discussing a cluster.

Accept several seconds between source commit and search visibility. An accepted source post need not appear in search immediately. Deletion and restriction are checked when results are returned, so an old index entry cannot grant access. Choose the recent two years as the interactive search scope and a slower path for older retained history.

Bound query length, page size and historical span. For this design, a missing required shard causes an explicit retryable search failure. Partial results could be a product option, but the API must then disclose incomplete coverage; a short list cannot silently claim to be exhaustive.

Clarify Boolean matching and sort order, how long indexing may lag, the interactive history window, and whether partial results are acceptable. The chosen requirements use newest-first results and an explicit failure when a required shard is missing.

02Functional requirements

Agree on these supported actions before selecting components.

1. Search public posts. Accept explicit AND/OR keyword queries and return matching current public posts in newest-first creation-time/ID order.

2. Page through bounded results. Let readers continue a query through a consistent ordered set of results for a limited browsing period; reject expired or mismatched continuation requests.

3. Apply source changes. Index creates, edits and deletions from the existing posting service, using source-assigned versions and compatible analyzers.

4. Disclose query failure. Return an explicit retryable error if a required shard cannot answer. Do not silently present incomplete shard coverage as exhaustive results.

03Non-functional requirements

Use these as illustrative interview assumptions to agree with the interviewer. Numerical targets require measurement; they are not claims about an existing product or a proven implementation. p95 (the 95th percentile) means 95% of measured operations finish within the stated time.

1. Workload and history. Use approximately 28,935 searches/s and 23,150 source changes/s at the planning peak. Interactive search covers two years; older retained history has a separate slower path.

2. Query latency. Target regional p95 response latency of 300 ms for the agreed bounded query mix at admitted peak load during normal operation. The benchmark must include common terms, shard merge and final source checks, not only cached rare-term queries.

3. Index freshness. Target p95 source-commit-to-searchable delay within five seconds during normal ingestion. Newly matching edits can be absent until indexed even when every returned record is current.

4. Result consistency. Use one pinned index snapshot for a page sequence, with a two-minute lifetime and at most 100 results/page. Recheck current deletion, visibility and matching text; the snapshot does not freeze permission.

5. Recoverability. Retain durable source snapshot/change-log coverage sufficient to rebuild a lost search shard. A search-process or index-node failure must not destroy the source; if required history is unavailable, stop and rebuild from a newer source snapshot.

6. Bounded and authorized work. Limit terms, historical span, candidate/refill work and page contexts. Return an error or shorter authorized page rather than unbounded work or uncertain private content.

04Retrieve matching IDs instead of scanning every body

An inverted index maps a term to the document IDs containing it. Each such list is a posting list.

Intersect the lists for AND, or union them and remove duplicates for OR. Then sort the matching IDs by creation time and ID, newest first.

One search process with a durable local search index is enough for the first version. The source stores a post and a pending change event in one transaction. A background indexer consumes that event, splits text into searchable terms and normalizes them using analyzer A1. An analyzer is the configured text-processing pipeline, and queries must use compatible rules.

After the index refreshes, the query can see T103. The search process retrieves its current source body and visibility, confirms it is still an eligible match and returns it. This final record-loading step is often called hydration. It prevents a candidate ID from being mistaken for permission to return stale cached text.

Use a mature search engine for the index files and posting traversal. The interview’s task is to explain the data path, updates and guarantees, not implement compression formats. This baseline is complete, but one process eventually runs out of useful storage, query CPU or ingest capacity.

05Measure posting work and distributed fanout

Assume 400 million posts/day averaging 300 bytes and 500 million searches/day. Use fifteen indexed terms per post as a storage estimate and fivefold peak traffic.

If each shard returns 100 candidates of 32 bytes, forty shards send 128 KB per query, approximately 3.7 GB/s at peak just for merging. By contrast, twenty returned 300-byte bodies total only 6 KB of raw text. Small result pages do not imply small retrieval work.

The five-byte document reference is an illustrative packed representation with enough range for this population, not a recommended permanent allocation format. Vocabulary strings are much smaller than all their postings. Measure common-word posting lengths, compression, deletion overhead and rebuild capacity before concluding the index fits in memory.

06Make query meaning and continuation explicit

Search request

GET /search?q=solar%20AND%20battery&sort=latest&limit=20

Reject malformed Boolean syntax instead of silently changing AND into OR. Authentication supplies quota and visibility scope.

A point-in-time snapshot retains a chosen visible index state for a bounded page sequence. The cursor binds that snapshot, normalized query, filters, sort definition and last returned sort values. Search-after pagination resumes beyond that last tuple. New posts therefore do not shift the numeric offsets of later pages.

Choose a two-minute snapshot lifetime and a maximum of 100 results per page for this exercise. Expired cursors require a restart. Reusing a cursor with another query is an error. Current deletion and access checks still run on each page: snapshot stability does not freeze permission.

The ingestion interface distinguishes source updates from search requests.

Source commit success and searchable visibility are separate API facts; report index freshness operationally rather than claiming that storing an event instantly updates every query replica.

07Preserve document identity across edits

Separate authoritative post records from the search index built from them.

Optional term positions support phrase search, but add bytes; the basic AND/OR scope does not require that feature.

Choose creation-time buckets, subdivided by a hash of post ID, for index ownership. All terms of one post live in the same document shard, and an edit stays in the original bucket. Moving a post whenever it is edited would complicate removal from its old location and duplicate suppression.

A body cache uses post ID and immutable content version. Query caches store candidate identities under their query, filters, sort and index version; cached candidates still undergo current checks. A post’s body and the permission decision must refer to the same version. Do not authorize public version one and then accidentally substitute an edited private version two.

Keep engagement features separate from lexical postings so every new like need not rewrite the text index. In the selected newest-first design, those features can be delayed display fields. A future exact most-liked sort must use them during candidate selection, not only rerank twenty recent matches.

08Partition by documents and prune by time

Document partitioning keeps the words and version of each post together. Each shard can evaluate the complete Boolean query locally, then return its newest candidates. The coordinator merges shard results by the same creation-time/ID order. Time buckets allow a last-day query to skip older indexes, while a broad historical query deliberately pays a larger fanout cost.

Term partitioning is a different option: solar might live on one machine and battery on another. It makes some single-term queries local, but common terms become hot and multi-term intersections cross owners. For the worked workload, choose time/document partitioning and accept bounded scatter/gather—the coordinator sends a query to relevant shards and gathers their answers.

Add replicas when query CPU saturates. They distribute independent queries but also consume index storage and ingest traffic. Select replicas that can serve the pinned snapshot. A replica being alive does not establish that it contains the required version.

Allocate a deadline for shard work that leaves time for merging and final record/visibility checks. Limit terms, candidates, refill rounds and page contexts. Under the selected strict-coverage policy, a required shard timeout fails the query; do not retry every shard repeatedly and multiply the overload.

09Apply newer document versions monotonically

Suppose T103 is edited at version 11 and deleted at version 12. If an indexer blindly applies a delayed version-11 event after the delete, it recreates a searchable old document. The writer must atomically compare the incoming version with the stored version and apply only a newer one.

A delete retains its version marker while older events can still replay. Simply removing the document and forgetting version 12 leaves no evidence that version 11 is obsolete. Repeated version-12 delivery has no additional effect. The index engine must prevent another update from intervening between the version check and the write, using serialized updates or an atomic version condition. A separate unlocked read is insufficient.

Persist source progress only after the corresponding index changes are recoverable. If an indexer applies an update and crashes before checkpointing, replaying it is safe through the version rule. If it checkpoints first and crashes before durable indexing, recovery may skip an update that never survived.

A source snapshot plus its matching change-log position allows a new index generation to be built, then caught up with subsequent events. Validate that generation before routing new searches to it. The precise multi-partition snapshot/watermark protocol is an Advanced follow-up, but the basic design must identify its recovery source and never claim a rebuild is current after incomplete replay.

10Return current matching content, then explain ranking

For solar AND battery, shards return matching candidate IDs and their sort values. The coordinator merges them, removes duplicates and batch-loads source records with current visibility. If T103 is now private or deleted, omit it. If it was edited to “new solar roof,” its current body no longer satisfies the query; re-evaluate the Boolean terms against that returned version and omit it until the index catches up.

This final check prevents incorrect disclosure and stale-text matches, but it cannot discover every newly matching edited post before indexing. Some newly matching posts and their correct ranking remain missing until indexing catches up. Fetch a bounded number of extra candidates after filtering; return a shorter page rather than loop indefinitely.

Matching and ranking are separate. Both T101 and T103 match the AND query, but newest-first favors T103. A lexical relevance score such as BM25 considers term frequency, corpus rarity and document length; it may favor the shorter, more focused T101. Scores from different shards need compatible analyzers and a defined corpus-statistics policy before they can be compared meaningfully.

Do not promise globally most-liked results by reranking only the newest twenty. A highly liked older match may never enter that candidate set. Either include popularity in each shard’s candidate selection using a pinned feature version or describe the result as a bounded-candidate approximation.

11Recover the index and bound expensive queries

The durable source and its retained change log recover lost search indexes. Restore a validated snapshot and replay from its recorded position. If the log no longer covers the gap, take a newer source snapshot instead of pretending a partial replay is complete. Keep capacity for temporary old/new index generations during analyzer changes and rebuilds.

Measure source-commit-to-searchable delay, query tail latency, posting entries scanned, shard fanout, timeout rate and rebuild duration. Check deletion behavior separately from ordinary indexing freshness. A fast response missing half the intended shard coverage is not a successful search under the chosen contract.

Rate-limit by authenticated client and query cost. Bound wildcards or omit them, cap terms and historical span, and cancel work after deadlines. Popular query caches can save repeated traversal, but arbitrary new queries still need real work. Current visibility failure causes omission or query failure, never optimistic disclosure.

Roll out analyzer changes into a separate generation and compare fixed examples for AND/OR behavior, languages, relevance and deletion. Removing stop words or changing normalization alters meaning; it is not merely a performance optimization. Search replicas and caches improve serving capacity, while durable source history and tested replay provide recovery.

12Check the design against the requirements

Use the agreed lists to check the finished design. The tests below still need to establish the targets; a proposed mechanism is not a measured result. FR refers to the numbered functional requirements above; NFR refers to the numbered non-functional requirements.

13Rapid revision

Remember: An index finds possible matches; retained versions stop old edits undoing deletes, and current source checks govern returned text.

Close by tracing T103 through durable source creation, asynchronous indexing, Boolean retrieval, shard merge and current-result checking. Then demonstrate delete version 12 defeating a late edit. The central tradeoff is efficient distributed search with a few seconds of index delay, while source records remain the basis for recovery and permissions. The next measurements are common-term traversal cost, fanout tail latency and rebuild duration.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.