Design a feature-flag and configuration platform

01Separate publishing a decision from making it

A feature flag changes application behavior without deploying a new binary. The control plane edits, validates and publishes rules; the evaluation path uses those rules on live requests. Keeping evaluation local can make it fast and available during a control outage, but means configuration may be temporarily old.

We use recommendation flag F7. It assigns a stable ten-percent tenant cohort to a new experience. Snapshot C17 is one complete saved configuration, and tenant 54 is the targeting key for the example. The flow is publish validated snapshot → distribute it → atomically install it in the application → evaluate trusted context → execute the branch → record bounded exposure information.

Clarify what the flag controls. Here it selects recommendations, not authorization, spending permission or an irreversible schema migration. A disconnected application may use the previous configuration for a bounded grace period, then must return false. An instantaneous universal kill switch would need a different online authority contract.

Start with a file and a deterministic evaluator on one instance. The engineering problem becomes interesting when many instances update at different times or operators publish incompatible rules. A dashboard containing an enabled boolean does not establish safe fleet-wide behavior.

02Functional requirements

Agree on what the service must do before choosing its components.

1. Author typed flags. Support boolean, numeric, string and structured flags, with separate development and production environments.
2. Target an experience. Select explicit users or tenants, percentage cohorts and multiple variants from trusted context; return the chosen value, reason and generation.
3. Publish and roll back. Validate and audit drafts/publications. A rollback publishes a new generation containing earlier intended values rather than moving the generation backward.
4. Inspect and retire flags. Show fleet adoption and bounded exposure information; retire obsolete configuration and application branches. The worked flag selects recommendations, not authorization, spending permission or irreversible schema changes.

03Non-functional requirements

Use these hypothetical requirements for the worked interview. Confirm the assumptions with the interviewer; the numerical targets require testing and are not measured results. For latency, p95 and p99 mean that 95% and 99% of measured delays, respectively, are no greater than the reported value.

1. Workload. Plan for 20,000 application instances and one billion peak evaluations/s. An SDK is the application library that loads and evaluates configuration; its actual runtime cost must be measured.
2. Latency and propagation. Target local p99 below 50 microseconds for bounded rules, publication p95 below one second and propagation to healthy connected instances within ten seconds p99. Publication success alone does not prove fleet adoption.
3. Deterministic, coherent results. The same snapshot and trusted context must produce the same result. Related flags in one request use one immutable snapshot; a delayed old download must not undo a newer generation.
4. Outage behavior. For F7, use the last validated configuration during a temporary disconnection. After five minutes without successful synchronization, return typed false with reason stale_config. This local timeout is not a strict five-minute global disable guarantee, and other flags may need different fallbacks.
5. Authenticity, access and privacy. Authorize environment writers and protect confidential rules/context. A checksum detects corruption; trusted transport or a signature under a trusted key establishes authenticity. Neither establishes that an authentic old snapshot is still current.
6. Bounded resource use. Limit rule complexity, targeting data, compilation and telemetry work. Exposure reporting must not block application evaluation, and old snapshots remain usable by requests already holding them.

04Make stable allocation and whole-snapshot replacement work

Randomly choosing ten percent on each request makes the same tenant bounce between experiences. Instead define a stable hash over an unambiguous tuple of rollout seed and targeting key. A hash deterministically maps input data to a numeric value. The seed is a fixed per-flag rollout value; keeping it stable keeps the same targeting key in the same cohort. In the example, bucket = hash(tuple(seed, key)) mod 10,000; modulo takes the remainder, giving a bucket from 0 through 9,999. Include buckets below 1,000 for ten percent.

Tenant 54 maps to 731 and therefore receives F7 = true. Increasing the threshold to 2,000 preserves that assignment if the seed, targeting identity and hash specification remain unchanged. Changing the seed deliberately reshuffles the cohort. Multiple variants need non-overlapping bucket ranges and an explicit assignment contract.

One instance loads complete C17, validates types and dependencies, compiles rules and stores an immutable active pointer. Each request captures that pointer, evaluates ordered explicit rules and then the percentage fallback, and returns diagnostics alongside the selected value. Related flags use the same retained snapshot.

A new file is parsed and compiled separately, then replaces the pointer atomically after validation. A malformed update leaves the prior version active. Updating individual entries in a shared mutable map could expose a combination never validated together. This baseline proves deterministic decisions and consistent replacement before introducing fleet distribution.

05Compare local work with distribution traffic

At 20,000 instances and 50,000 evaluations/s each, a remote call for every evaluation would create one billion network calls/s. Even a 100-byte combined envelope implies 100 GB/s before transport overhead and puts network latency on each application path. Local evaluation concentrates network work on relatively rare configuration updates.

Bound rule complexity and precompile expensive structures outside requests. Large targeting lists and unbounded expressions can dominate evaluation. Retaining old and new compiled snapshots during rollout costs memory, but permits in-flight requests to finish consistently. Measure publication bandwidth, compilation work and telemetry separately instead of assuming low lookup latency solves the whole service’s cost.

06Give drafts, generations and context distinct meanings

Edit a draft

PUT /prod/draft

Publish a validated draft

POST /prod/publish

Draft versions protect editors from lost updates; publication generations order committed releases. An optimistic version conflict asks the operator to review newer state rather than overwrite it.

Local evaluation call

getBoolean(F7, false, trustedContext)

Missing flags, initialization failures and type mismatches follow documented fallback behavior. Define attribute types, merge precedence and missing-value handling so different SDK languages cannot interpret the same context differently.

The targeting key identifies the assigned user or tenant. Here all members of tenant 54 share one tenant-based assignment. The application derives the active tenant from authenticated membership, not an arbitrary request field. Country or plan can be additional typed rule inputs.

Control records

A publication changes the active pointer only after its immutable bytes are durable. Notifications are recoverable announcements of that pointer, not the only record that a publication occurred.

07Identify the limits of the one-instance design

The first limit is operational: many editors and environments cannot safely overwrite files without version checks, audit and validation. The second is distribution: 20,000 instances can miss notifications, download at different speeds or disconnect. Publishing centrally does not prove a particular application is already using the new value.

Consider app8 starting a C17 download. C18 arrives and installs first; C17 then finishes. “Last download completed wins” moves the instance backward. A genuine rollback to C16’s content must be published as generation 18, so the same ordering rule works for ordinary updates and intentional reversals.

A partial-map race is different: a request reads F7 from the new configuration and dependency F8 from the old one. Locking each flag independently does not give the request one consistent configuration. The request must retain one whole snapshot across related evaluations.

Finally, app9 is partitioned while an operator disables F7. No push protocol can deliver information across a broken connection instantly. The product must choose how long old behavior may continue and what follows. These failures motivate versioned publication, recoverable distribution, atomic installation and freshness policy as separate mechanisms.

08Add a control plane and recoverable delivery

Publication sequence

1. Authorize and validate. Authenticate the environment writer; check rule types, bounds, missing references, dependency cycles and complexity.
2. Store the snapshot. Make the complete immutable bytes durable.
3. Publish conditionally. Commit the active-generation pointer and audit/publication event only if the expected generation still matches.

A scheduled publication revalidates its assumptions at activation rather than blindly applying an obsolete draft.

A stream announces that a newer generation exists. Instances fetch immutable content through regional caches and periodically poll the authoritative active pointer to recover missed announcements. The notification prompts a check. Cached bytes provide the configuration, but only the publication service can confirm which generation is current. This keeps distribution efficient without relying on uninterrupted streaming.

Installation sequence

1. Verify. Check environment, schema, checksum and trusted origin/signature.
2. Compile. Prepare the fetched rules outside the request path.
3. Compare and install. Atomically replace the current pointer only when the incoming generation is newer; ignore older/equal generations.
4. Handle a race. If another installer changes the pointer, retry the comparison against that current value.

Application requests continue using their captured references until completion. Cleanup waits for those references before reclaiming old compiled structures. The cost is temporary duplicate memory and careful concurrency. Local evaluation remains independent of telemetry: aggregate and bound exposure reporting so a metrics outage does not become a recommendation outage.

09Choose a practical stale-configuration policy

A signed C17 can remain authentic after C18 is published. A successful download proves that a configuration is available, not that the fleet has installed the latest publication. Keep those observations separate in the SDK and the operations dashboard.

For this recommendation experiment, the practical policy is last-known-good operation during a short outage. Periodic synchronization reads the current environment generation and fetches a replacement when needed. Track successful synchronization with monotonic elapsed time. After five minutes without it, return false and report stale configuration. Cached bytes after a restart do not establish a new successful synchronization.

This policy reduces disruption without claiming an instantaneous kill switch. A request already using C17 can finish with its captured decision. Different application instances may briefly disagree during propagation. A delayed reply may leave an application unaware of C18. The five-minute local timeout therefore does not prove that every instance disables F7 within five minutes of publication.

If a feature must stop sensitive actions immediately, check a current online authority at that action boundary and accept its latency and outage behavior. A strictly bounded global stale-use promise requires additional timing and renewal rules. Keep that stronger protocol in the advanced discussion; ordinary recommendation rollout does not need it.

10Prove rollback and request consistency with C17 and C18

An operator validates F7 against draft 16 and publishes C17/generation 17. app8 fetches, verifies and installs it. Request req61 derives tenant 54 from trusted context, retains C17, evaluates bucket 731 below 1,000 and uses the enabled recommendation branch. It records an exposure only if the decision actually influenced the experience.

Now generation 18 contains the intended older disabled values. If its download finishes before 17, installation leaves 18 active and ignores the later 17 result. If 17 finishes first, 18 replaces it afterward. Both orders converge to 18. An existing request retaining 17 may finish consistently under 17, while the next request selects 18; the documented stale-configuration policy still applies.

If C19 is malformed or uses an unsupported schema, validation fails before pointer replacement and 18 stays active. If two publishers race, the expected-active-generation check permits one transition and gives the other a conflict. Restoring an earlier boolean value is still a new publication, so its generation must increase.

Exposure data names flag, variant and generation actually used. Assignment, evaluation and exposure are different observations. A debug evaluation or hidden branch should not automatically count as a user seeing the experimental experience.

11Measure fleet adoption and bound configuration risk

Observe publication-to-install lag, active-generation distribution, time since successful synchronization, typed fallback rate, compilation failures, evaluation latency and outcome metrics by cohort. A healthy fleet average can hide an enabled cohort with errors. A rollback is not complete merely because the control API returned committed; inspect adoption and recovery of the intended product metric.

Use the same fixed input/output test vectors in every supported SDK language. Test seed and targeting-key migrations intentionally because either can reshuffle users. Protect environment credentials and audit production publications. Browser clients must not receive confidential rules or other tenants’ attributes simply to evaluate a flag; server-side evaluation can keep those details inside a trusted boundary.

Drill concurrent editors, reversed download order, malformed configuration, disconnection, restart with old bytes and delayed synchronization responses. Roll compatible code/data changes separately; a flag cannot undo irreversible writes or make an old schema valid for new code.

Retire flags after rollout by removing obsolete application branches as well as configuration. Experiments additionally need stable assignment, genuine exposure and statistical analysis. A percentage slider supplies controlled assignment, not proof that a treatment improves outcomes. Revisit remote evaluation when current central authority or confidential rule access outweighs local latency and offline continuity.

12Check the design against its requirements

Before closing, check the final design against the agreed requirements. FR means functional requirement and NFR means non-functional requirement; the numbers refer to the lists above. These are proposed validation checks, not test results.

13Rapid revision

Rehearse the numbered functional requirements and non-functional targets first. Use this table to recall the mechanisms, then close with the requirements check above.

Remember: Newer generation wins, not the last download.

Close with: “I keep publication audited and evaluation local. Stable cohorts make rollouts repeatable, whole snapshots prevent mixed rules, and monotonic installation handles download reordering. A documented last-known-good policy and local outage timeout govern stale use. I accept bounded offline fallback and measure actual fleet adoption rather than calling a central write an instantaneous global switch.”

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.