Design a metrics, logging and tracing platform

01Start with an incident the platform must explain

Checkout request req81 becomes slow and times out while calling inventory. The platform should let an operator see increased checkout latency, locate the timeout log and inspect the trace linking checkout to inventory. Metrics describe aggregate behavior, logs describe individual events, and traces connect timed operations called spans across a request. These signals complement one another rather than being three interchangeable storage formats.

Build operational monitoring with bounded queries and explicit loss reporting. Required financial or security audit records need a separately specified zero-loss or durable-admission contract; ordinary debug telemetry must not silently inherit that promise. Include ingestion, dashboards, structured log search, trace lookup, alert rules and retention. Exclude arbitrary joins over every event ever collected.

Target recent accepted metrics becoming visible within thirty seconds for 99% of samples under normal load and target bounded dashboard queries finishing within two seconds p95. Durable ingestion acceptance is a separate milestone from search visibility. A trace may be sampled or incomplete; a missing span does not prove the service was never called.

An empty graph during a collection outage is missing evidence, not a measured zero. The interface and alert engine must show stale or incomplete data instead of declaring checkout healthy.

Ask whether this is operational telemetry or mandatory audit evidence, which incident queries must be fast, and how much collection loss is acceptable during an outage. Choose operational metrics, logs and traces here with reported dropping of telemetry before acceptance, durable central acceptance and bounded recent-data queries.

02Functional requirements

1. Collect diagnostic signals. Ingest metric samples, structured logs and linked trace spans with tenant identity and useful request correlation.

2. Investigate incidents. Provide recent dashboards, bounded structured-log search and trace lookup, showing freshness, sampling and missing evidence.

3. Evaluate alerts. Evaluate versioned rules over suitable data, preserve pending/firing transitions and notify operators of meaningful state changes.

4. Manage history. Retain and query the selected recent history, support documented archive/downsample policies and expose rejected or lost telemetry.

03Non-functional requirements

These are illustrative interview assumptions, not product facts or measured benchmarks. Confirm them before choosing components, then validate the completed design under the stated workload. Here p95 means the 95th-percentile latency: 95% of measured requests take no longer than that value. Report errors and rejected work alongside latency; a fast failure is not a successful outcome.

1. Ingestion and retention. Plan for ten million active series sampled every fifteen seconds, about 666,667 samples/s, plus 100,000 logs/s at 500 bytes each. Use thirty days of metric retention and seven days of indexed logs for the estimates; sampling and archive policies are separate choices.

2. Visibility and query latency. Target recent accepted metrics becoming queryable within thirty seconds for 99% of samples under normal load, and bounded recent dashboard queries completing within two seconds p95. Disclose slower archive queries and measure timeout/rejection rates alongside latency.

3. Loss and recovery boundary. Before central acceptance, collectors may shed ordinary telemetry only under a bounded, visible policy. After acceptance, preserve retained records through API/writer restarts and replay them; configure and test the chosen backend’s replica failure guarantees explicitly.

4. Diagnostic correctness. Compute reset-aware rates and compatible histogram aggregation; do not interpret missing samples as zero or incomplete traces as proof an unobserved call never happened.

5. Isolation and privacy. Enforce tenant, cardinality, byte and query-work limits. Redact sensitive payloads before persistence and prevent historical queries or unbounded collector buffers from exhausting current application/ingestion capacity.

04Complete the path from request to diagnosis

Start with instrumented applications, a local collector, a time-series engine, a structured log store, a trace store and a query/alert service. OpenTelemetry-style instrumentation can attach request and trace IDs consistently. The collector batches records so each application request does not open its own telemetry network connection.

For req81, checkout increments a request counter, records its duration in a histogram and emits a structured error log with the trace ID. The collector submits a bounded batch. The receiving store acknowledges according to its configured durable-write policy; the collector can then release that accepted batch from its retry buffer.

The dashboard reads a recent bounded interval for checkout. An alert detects a sustained error ratio, records its pending/firing transition and links to the evaluated time range. The operator opens the trace, sees time spent waiting for inventory and searches logs by the same request ID. This is a complete useful product before a custom streaming storage system is added.

The baseline includes tenant authorization, redaction and buffer limits. If telemetry collection consumes unlimited memory or blocks application requests indefinitely, a monitoring outage can cause the checkout outage it was intended to diagnose. State from the beginning which records a full collector may drop before service acceptance.

05Define metric identity and aggregation semantics

A time series is a sequence of measurements with one metric name and label set. For example:

One checkout error-counter series

requests_total{
  service="checkout",
  instance="c3",
  status="500"
}

Changing the instance or status label identifies a different series. A counter increases as events occur and may reset after restart. A gauge reports a current value such as queue depth. A histogram records a distribution through counts in value ranges.

If one instance's counter restarts from 1,000 to 5, summing raw counters before handling resets creates misleading drops. Likewise, averaging two instance p99 values does not produce the service p99: the summaries omit the underlying distribution and request volumes. Histogram bucket boundaries determine approximation error, so use compatible boundaries or a backend-supported compatible histogram representation.

06Count series identities as well as payload bytes

Assume ten million active series sampled every fifteen seconds. That produces about 666,667 samples/s. At an illustrative sixteen bytes for a scalar timestamp/value pair, raw payload is 10.67 MB/s or 0.922 TB/day. Thirty days with three copies is about 83 TB before compression, indexes and metadata. Histograms may require multiple bucket series or larger native samples, so their actual encoding must be included separately.

Logs at 100,000 events/s and 500 bytes each produce 50 MB/s or 4.32 TB/day. Seven days is 30.24 TB before replication and indexing. Indexing every arbitrary field can multiply the storage cost; select fields used by the promised incident queries.

Cardinality is the number of distinct series. A family spanning 100 services, twenty regions, ten statuses and fifty instances could create one million series if every combination exists. Adding request IDs or user IDs can create a new series for nearly every request. A byte limit alone does not protect the series dictionary from that churn.

Set per-tenant limits on active series, new series per minute, log bytes and query work. Measure real label distributions and compression rather than treating a theoretical maximum or raw payload estimate as a final machine count.

07Expose identity, time and completeness

Signal records

Example structured error-log identity and timing

{
  "eventId": "log-req81-1",
  "service": "checkout",
  "eventTime": "2026-10-07T09:00:00.000Z",
  "ingestionTime": "2026-10-07T09:00:00.200Z",
  "severity": "ERROR",
  "requestId": "req81",
  "traceId": "trace-81"
}

Ingestion and query contracts

Event time is when the application observed something; ingestion time is when the platform accepted it. Both matter: an old log may have been buffered by its source rather than delayed by central storage. Specify the allowed out-of-order window and route later data to a defined correction or archive path.

Retried batches retain stable record identities. Two legitimate logs with identical timestamps are still different events. For a simple log store, unique tenant/event IDs make repeated insertion harmless; metric sample identity follows the selected backend's documented series/time and conflict rules. Reject conflicting reuse rather than silently counting a repeated record as new.

Responses must make partial acceptance explicit so the collector retries only the appropriate records. Bound the deduplication/retry horizon and account for its metadata rather than promising infinite remembered identities.

08Add a durable buffer and isolate query resources

If storage outages or bursts exceed direct-ingestion capacity, insert a replicated ingestion log between authenticated gateways and the signal backends. Gateways validate tenants and limits, then acknowledge only after the selected durable append policy. Writers consume accepted records into the time-series, log and trace stores. Search visibility can now lag while accepted data remains buffered.

Use established backends and their supported ingestion/replay integration rather than designing a new chunk-publication protocol during the first interview. Where duplicate-free storage is promised, backend writes must recognize stable record IDs when writers retry. If a backend can expose duplicates, disclose that behavior and deduplicate the relevant query rather than claiming the broker alone supplies exactly-once storage.

Partition metrics by tenant and series identity, logs by tenant/time plus a distribution key, and trace lookup by trace ID. A large tenant needs several partitions; one tenant label must not force all its work onto one machine. Within each backend partition, preserve its required record order and allowed lateness.

Separate expensive historical queries from current ingestion through dedicated resource pools and scanned-byte/concurrency limits. A user investigating an incident should not make every current alert stale by launching an unlimited seven-day search on the same disks.

09Design the questions before indexing everything

For a checkout dashboard, first select series using low-cardinality labels, then read only the requested time interval and resolution. Apply counter resets before aggregation and combine histogram distributions correctly. Return the data-through time and disclose missing partitions. A partial result must not look like a complete service-wide graph.

For req81, narrow by tenant, service and event-time range, then use indexed request or trace IDs. Return ingestion time so the operator can identify delayed evidence. Log pagination needs a stable tie-breaker such as event ID and either a fixed query snapshot or declared live-search behavior. A cursor alone does not freeze incoming records.

A trace view assembles spans under the trace ID and shows missing relationships. Head sampling decides near request start and propagates the decision; tail sampling waits for spans and can favor error traces, but costs buffering and cannot guarantee all late spans arrive before its decision. State sampling rates and incompleteness in the UI.

Cache repeated bounded dashboard queries using tenant, filters, resolution and the relevant freshness boundary. Late data may require invalidating or refreshing affected ranges. Downsampling saves storage but permanently removes time detail; retain count and sum for means and compatible histogram information for distribution queries.

10Make alert state depend on fresh evidence

An alert evaluator might run every fifteen seconds and fire when checkout's error ratio stays above a threshold for a specified duration. Record the rule version, last evaluated boundary, pending start and current state. This prevents a restart from inventing a new alert every evaluation or forgetting all pending progress.

The evaluator first checks whether the input is sufficiently fresh and complete. Missing data produces a stale or unknown state, not a zero-error rate and automatic resolution. A previously firing alert should follow an explicit missing-data policy; silence is not proof that the incident ended.

To fire after a threshold holds continuously, the evaluator needs observations covering the whole interval. If the platform was unable to observe a five-minute interval, retaining an old pending timestamp does not prove the threshold held throughout it. Reconstruct that interval from retained samples if possible; otherwise restart the duration check according to the rule rather than counting unseen time as demonstrated failure.

Persist state transitions and use stable notification identities so retries do not create repeated pages for the same transition. Include the evaluated query and time range in the notification. Monitor the observability platform with an independent small heartbeat or external probe; relying only on its own failed ingestion path can conceal its complete outage.

11Bound buffers and describe the acceptance boundary

The before/after acceptance distinction is essential. Before central acceptance, configured collector buffers can overflow and ordinary telemetry may be lost. After acceptance, the platform has promised retention under the configured backend durability policy, including survival of API/writer restarts. It must not overwrite unprocessed records merely because the buffer is full. Tighten admission before that happens.

At 100,000 log events/s, a one-hour outage accumulates 360 million events. A recovered writer handling 150,000/s while 100,000/s new traffic continues drains at only 50,000/s, taking two more hours. Reserve recovery capacity and measure oldest unprocessed age. Returning to normal ingest throughput alone does not catch up.

12Control cost without destroying incident evidence

Apply tenant-scoped ingestion credentials, label allowlists and redaction before persistence. Private data in labels both leaks information and creates expensive series identities. Restrict retention and alert-route changes, and avoid raw payloads in diagnostic labels about the platform itself.

Monitor accepted-to-visible delay, collector drops, series churn, writer backlog age, query scanned bytes, alert freshness and backend errors. A fast ingestion endpoint can still feed a platform whose dashboards are hours behind. The independent heartbeat should exercise a known write and read, not merely check that a process responds.

Retain recent high-resolution metrics and indexed logs on fast storage, then archive or downsample according to the actual investigative needs. Reducing seven-day indexing can save substantial cost but may make old request searches slower or unavailable. Show those limits rather than promising the same query latency across every retention tier.

Test lost batch acknowledgments, duplicate event IDs, counter resets, incompatible histogram changes, collector overflow and backend replay. During backend upgrades, compare counts, sums and histogram buckets on representative traffic. A custom immutable-block engine would also need to prove that publication and cleanup cannot expose duplicate or missing blocks; selecting a proven backend lets the first interview focus on signal correctness and the complete operating contract.

13Check the design against its requirements

Use the numbered requirements to check the final design. FR refers to the functional list; NFR refers to the non-functional list. Performance rows specify tests still required, not achieved benchmark results.

14Rapid revision

Remember: Missing samples cannot prove zero errors. Check freshness before an alert declares recovery.

Close with req81: a metric reveals the symptom, a trace identifies the slow dependency and a log explains the timeout. Then fail the collector and show the dashboard becoming stale. This demonstrates both the diagnostic product and the limits of its evidence.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.