Design a recommendation platform

01Define usefulness before choosing a model

A recommendation service selects a small set of items from a much larger catalog. For this interview, it returns twenty videos with display metadata. Video storage and playback are separate services. The objective is useful viewing and satisfaction, subject to safety, diversity and latency requirements; maximizing clicks alone can reward misleading thumbnails or repetitive content.

Ask what user outcome defines a useful recommendation, which eligibility rules are mandatory, and whether anonymous or opted-out viewers must still receive results. The worked design serves contextual recommendations to those viewers and treats safety and access as hard constraints.

Use viewer U7 opening the home page. The catalog contains ten million videos, but the response needs only twenty. U7 has recently watched cooking lessons and already completed item I12. The service must find plausible candidates, remove unavailable or inappropriate items, rank the remainder and return a varied set. Feedback later describes what U7 actually saw and watched.

The complete flow is request context → candidate retrieval → eligibility checks → ranking → diversity rules → response → feedback. Retrieval narrows the search space; ranking compares the selected candidates in more detail. These are different jobs. Begin with regional popularity so the service works before personalization, embeddings or a trained model exist. Learned components should improve a measured baseline rather than become unexplained prerequisites.

02Functional requirements

Agree on what the service must do before choosing its components.

1. Serve a recommendation page. Return up to twenty permitted videos with display metadata for authenticated or anonymous viewers. If fewer eligible items exist, return fewer rather than violate a content rule.
2. Respect viewer context. Apply language and region constraints, completed-item filtering and explicit dismissals; support cursor-based continuation bound to the viewer/session and feed context.
3. Support personalization choices. Use permitted history for personalized retrieval/ranking and offer contextual popularity after opt-out or for a cold-start viewer.
4. Capture useful feedback. Record actual visibility, clicks and watch activity with response and event identities. These observations support quality measurement and later ranking changes; video storage and playback are separate services.

03Non-functional requirements

Use these hypothetical requirements for the worked interview. Confirm the assumptions with the interviewer; the numerical targets require testing and are not measured results. For latency, p95 and p99 mean that 95% and 99% of measured delays, respectively, are no greater than the reported value.

1. Workload. Plan for a ten-million-item catalog, 5,000 peak recommendation requests/s and 1,000/s on average. Bound the candidates scored for each page rather than examining the full catalog.
2. Latency and availability. Target response p95 below 200 ms and 99.9% availability with the chosen candidate count and feature access pattern. Degraded personalization may fall back to eligible contextual results within that budget.
3. Content eligibility. Check the current policy available at the defined serving boundary. A high score never overrides a hard rule; if eligibility cannot be established, omit the item or fail appropriately. Later removal cannot recall an already delivered response, so playback needs its own current checks.
4. Recommendation quality. Evaluate useful watching and long-term satisfaction alongside clicks, abandonment, repetition and safety violations. Define experiment metrics and guardrails before a rollout: higher clicks with worse completion or more complaints may be a regression.
5. Consent and privacy. Opt-out disables history-based retrieval/ranking and invalidates personalized continuation state. Limit personal-history retention and specify how deletion reaches derived features, including its propagation delay; do not let an old cached page silently continue withdrawn personalization.
6. Trustworthy observations. Separate returned items from actual impressions, validate feedback and count a retried event once within the supported replay horizon. Feedback delay may make features older; it must not be hidden as current complete data.

04Serve a useful popularity list end to end

Maintain a catalog containing item ID, region availability, language, publication status, creator and basic popularity. A scheduled job computes popular items per region and language using recent verified activity. The serving API retrieves the top two hundred for U7’s context, checks current eligibility and removes completed or explicitly dismissed items.

Sort the remaining candidates by the baseline score, then select twenty while limiting repeated creators. Return their IDs, display metadata and a response ID. Persist or otherwise durably capture enough response context to interpret subsequent feedback: which items were returned, their positions and the ranking version. The UI reports actual visibility separately from the server’s decision to return an item.

The feedback endpoint accepts a stable event ID and verifies the viewer/session and response context. A durable queue retains events for aggregation. Retried copies of an event must not become several views or watch sessions.

This baseline needs an API, catalog, popularity job and feedback pipeline. It already handles cold-start users and can remain the fallback later. Its weakness is relevance: two people with different interests in the same region see similar candidates. That observed limitation, rather than fashion, motivates personalization.

05Bound online work before adding model complexity

At five thousand peak requests/s, scoring every one of ten million items would require fifty billion item scores/s. Even a 50-microsecond score would consume roughly 2.5 million CPU-seconds each second. This arithmetic explains why candidate retrieval is necessary, not merely an optional optimization.

If retrieval narrows to two hundred candidates per request, the peak is one million scores/s. At the same illustrative cost, that is fifty fully busy CPU cores; at fifty-percent utilization, roughly one hundred before redundancy. Feature lookup, filtering and serialization still consume budget. Benchmark the whole path rather than treating this multiplication as a deployment guarantee.

Ten million 128-dimensional float 32 vectors contain about 5.12 GB of raw values. A searchable index needs additional memory for its data structures, metadata and replicas. An embedding is a learned numeric representation that places related items near one another; raw vector storage is not the total cost of searching them.

At one thousand average requests/s and twenty returned items, the service returns 1.728 billion item positions/day. If every position produced a 300-byte event, that is about 518 GB/day before replication. Actual visible impressions differ, so use this as a workload estimate, not a claim that every returned item was seen.

06Preserve response and event identity

Recommendation request

GET /recommendations

Recommendation response

A continuation token must not let one user adopt another user’s personalized page.

Feedback event

POST /feedback

Validate feasible durations and ordering without assuming every disconnected client uploads immediately.

Catalog and feature records

These definitions keep the ranker from mistaking milliseconds for seconds or missing data for a measured zero.

Keep model version and compatible feature definitions in a deployment bundle. The bundle is a named combination checked before rollout; naming it alone does not prove compatibility. Feedback names the served version so offline analysis can explain which policy produced an outcome.

07Add personalized sources without losing the fallback

A regional popularity list can miss niche cooking videos that match U7’s interests. Add several bounded candidate sources: recent videos from followed creators, matching topics, similar items and regional popularity. Each source returns a limited set. Merge duplicate item IDs, check basic eligibility and cap the combined pool before expensive feature enrichment.

Similarity retrieval can use an approximate nearest-neighbor index over item embeddings. Approximate means it trades exhaustive comparison for faster search and may miss some mathematically nearest items. The goal is a useful candidate pool, not a proof that a vector neighbor is the best video. Evaluate retrieval separately from the final ranker.

For example, gather at most one thousand candidates, then retain two hundred for richer scoring using a cheap preliminary score and source quotas. The exact limits are tunable budgets. A source that returns nothing should not stall the entire page; popularity remains available.

New users rely on declared interests and context. New items need metadata-based retrieval or a controlled exploration allocation because they lack watch history. Popularity-only feedback can otherwise keep them invisible forever. Exploration accepts some uncertainty to collect evidence, subject to the same safety and eligibility rules as ordinary recommendations.

08Rank a bounded set and apply product constraints

Fetch the two hundred candidates’ features in batches, not one network round trip per item. A simple explainable score can start with:

score = 0.60 × interest + 0.25 × quality + 0.15 × freshness

Each input is normalized to the intended zero-to-one scale. These weights are illustrative product choices, not universal constants.

Candidate example

A hard content rule is not just a small negative weight.

A trained ranker can replace the hand-set formula after offline evaluation and controlled online testing. It predicts an explicitly defined target from the same documented features. Missing features use trained or specified defaults; incompatible features trigger a fallback rather than arbitrary interpretation.

Finally apply creator caps and diversity rules to the ranked list. This may lower the raw predicted score while improving the overall experience. Select the page under those rules, even when it differs from the twenty highest scores.

09Keep training away from the request deadline

Scale the serving API horizontally and cache reusable item metadata and popularity lists. Personalized feature access uses bounded timeouts and privacy-aware keys. The online path retrieves candidates, batch-loads features, scores a bounded set, checks eligibility and returns the response. It does not train a model while U7 waits.

A separate pipeline consumes feedback, deduplicates stable event identities and updates aggregates. Scheduled training joins examples with feature values appropriate to the event’s historical context. Using information that became available only afterward leaks the future into evaluation and can make an ineffective model appear excellent offline.

Deploy a checked model/feature/index combination gradually. Canary traffic and an easy switch to the popularity baseline limit the impact of a bad release. An item index and model may have different update cadences, but their representation and schema compatibility must be explicit.

Keep queue lag and feature age visible. Durable events may arrive late, so freshness and completeness are different properties. If one retrieval source or feature service times out, use a documented simpler path within the response budget. The fallback still applies current eligibility checks; degraded relevance does not authorize unsafe content.

10Interpret observations before learning from them

Returning I11 at position seven does not prove U7 saw it. The application may display only the first screen, the request may be abandoned or the item may be hidden. Record assignment, actual visibility, click and watch as distinct event types. This gives training and experiment analysis a chance to distinguish opportunity from outcome.

A retried visibility event reuses its original event ID. The consumer’s deduplication and aggregate update must be coupled, for example in one transactional sink, so a crash does not count the same observation twice. Retain deduplication state for the supported replay horizon; a key forgotten too early cannot protect a later retry.

An absent click is not automatically a negative preference when visibility is unknown. Likewise, a ten-second watch means different things for a twelve-second clip and a two-hour lecture. Define target labels with product context rather than training directly on whatever telemetry happens to be easiest to collect.

Use stable experiment assignment and record the policy actually served. Compare satisfaction, safety and latency guardrails in addition to the target metric. Position bias and exploration make causal conclusions harder; detailed counterfactual methods belong to advanced analysis, not an unsupported claim that raw clicks reveal pure preference.

11Recover serving and test the learning loop

Serving failure choices

Trace U7’s page through request, response ID, visible impression and watch event. Then repeat the feedback event, delay it, remove I11 before a later request and disable personalization. Verify one counted event, documented late-data behavior, no newly served removed item under the policy boundary and a contextual response after opt-out.

Measure candidate recall on judged examples, ranking quality, response latency, feature age, queue lag, duplicate rate, creator concentration, cold-start coverage and outcome guardrails. A good offline ranking metric does not by itself prove user benefit; validate with a controlled rollout.

Protect feedback APIs from fabricated events and limit retained personal history. Group monitoring metrics by bounded categories rather than creating a new metric series for every user. The next investment should follow evidence: poor retrieval needs better candidates, slow feature reads need serving work, and misleading labels need measurement repair rather than a larger model.

12Check the design against its requirements

Before closing, check the final design against the agreed requirements. FR means functional requirement and NFR means non-functional requirement; the numbers refer to the lists above. These are proposed validation checks, not test results.

13Rapid revision

Rehearse the numbered functional requirements and non-functional targets first. Use this table to recall the mechanisms, then close with the requirements check above.

Remember: Returned is not seen; observation drives learning.

Close with: “I first serve a complete popularity feed and measure it. Candidate retrieval makes richer ranking affordable, while eligibility and diversity remain explicit product rules. A separate feedback and training pipeline improves relevance, and the original baseline remains an operational fallback.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.