Production readiness: SLI, SLO, observability, and recovery

01What is production readiness?

Production readiness means being prepared to run the service through ordinary traffic, changes, overload, and failures. First define what users must be able to do and how reliably and quickly the service must respond. Then decide how to measure whether it meets those requirements and how to recover when it fails. Observability is the ability to understand internal behavior from the metrics, logs, and traces that the service produces.

An SLI (service-level indicator) is a quantitative measure of service behavior, such as the fraction of photos ready within 60 seconds. An SLO (service-level objective) is a target for that measurement over a window. An SLA (service-level agreement) is a commitment with agreed consequences, often contractual; it is not simply another name for an internal SLO. An error budget is the amount of failure the SLO permits over its measurement window, such as the number of requests allowed to miss a completion deadline.

Measure whether the service finishes the operation the user requested, including any background processing required before the result is usable. In the example, upload P501 is accepted immediately, spends 95 seconds queued, takes four seconds to render and one second to publish, and becomes ready after 100 seconds. That event misses a 60-second completion threshold despite a successful acceptance response and running API processes.

Check the complete user operation, protect the resources it needs, deploy changes safely and test recovery. A running process is useful evidence, but does not prove the service is fast enough, saves data correctly or enforces access permissions.

02SLI, SLO, SLA, and error budget: definitions and calculation

A service-level indicator, or SLI, is the measured behavior. A service-level objective, or SLO, is its target over a stated window. For completion, the numerator counts eligible photos ready within 60 seconds. The denominator counts eligible accepted photos whose evaluation period has elapsed. A just-accepted photo cannot be labeled late before its 60-second allowance ends.

Calculate the error budget

For one million evaluated photos, the 0.1% allowance permits at most 1,000 bad completion events. This allowance is an error budget. P501 is one bad completion event that consumes this budget; one late photo alone does not prove the aggregate 30-day 99.9% SLO was violated. Specify whether unsupported file types, canceled uploads, and failures caused by our service count. Exclusions should reflect the contract, not hide inconvenient incidents. Availability, timely completion, and correctness can require different indicators.

Define exactly which events enter the window

Handle no traffic and missing telemetry

When there are zero eligible events, the ratio is undefined, not 100% healthy. Use a no-data signal and the separate acceptance indicator or synthetic check. A time-based 99.9% availability target over 30 days permits 43.2 minutes of bad time, but that is a different denominator from the one-million-photo event budget. Do not convert between them without traffic assumptions.

A synthetic check performs a controlled test operation, such as uploading a test image and verifying that it becomes viewable. It can reveal a broken path when real users are inactive. Report that test separately from the real-user completion ratio rather than using it to invent a denominator for a no-traffic period.

03Observability and the four golden signals

Metrics are numerical measurements over time. For this service, track upload demand, timely completion, queue age, worker capacity, and errors. The classic four signals are latency, traffic, errors, and saturation: how long work takes, how much arrives, what fails, and which resource is nearly full. Google SRE monitoring.

The 100-second completion is the symptom. High queue age tells us where to investigate; it is not yet the cause. CPU may be low because a worker-concurrency setting is too restrictive, not because there is no demand.

Break down metrics by processing stage and software version. Control label cardinality: the number of distinct label values and combinations that create separate time series. A separate time series for every photo ID would be costly; IDs belong in targeted event records and traces.

04Logs and distributed traces: locate the missing 95 seconds

Logs record individual events; structured fields make those records searchable. Traces connect work across stages so we can follow one request or asynchronous job. A span records one timed operation within a trace, such as a database call or a worker processing a photo. Carry a correlation identifier that links records for the same job without exposing secrets or personal data through acceptance, queue delivery, rendering, and publication. For asynchronous work, preserve the relationship even when it is represented by a trace link rather than one continuously open call.

Rendering took four seconds and publication one. Almost all delay was before work began. We inspect the new worker release and discover that its concurrency limit was unintentionally reduced. That mechanism fits both the queue wait and low CPU.

Logs must not copy private image contents, access tokens, or unnecessary personal data. A photo ID and authorized diagnostic lookup are usually more useful than dumping the entire payload into an unrestricted log.

A practical implementation can instrument request and worker spans with OpenTelemetry, propagate trace context in the job metadata, and export selected traces and structured logs to a backend. Use the durably stored job record to decide whether a job completed; sampled traces are diagnostic evidence, not a complete SLO denominator. Cross-host timestamps may differ, so record stage durations with monotonic timers, which measure elapsed time without jumping when the system clock is adjusted, and account for clock uncertainty when subtracting timestamps from different machines.

05Actionable alerts and error-budget burn rate

A dashboard helps investigation; an alert asks someone to act. Paging on every brief CPU spike creates noise and does not necessarily protect the completion objective. Tie urgent alerts to significant user-impact or rapid budget consumption, with enough evidence to identify the affected service and likely response.

Suppose a recent window has 2% late photos while the SLO allows 0.1%. The burn rate is 2% / 0.1% = 20: the service is consuming its error allowance at twenty times the reference rate under that measurement. Use both shorter and longer windows so a severe ongoing problem is detected without treating a tiny transient sample as a sustained incident. SLO alerting reference.

Also monitor correctness constraints. A timely response that exposes a private photo is not a successful product outcome. Audit access-control decisions and check that rules such as “only authorized users can view a private photo” hold; latency metrics cannot establish confidentiality. The security-and-multi-tenancy chapter explains where and how to enforce those access checks.

For the illustrative 30-day window, a sustained 20× burn would consume a full window's budget in about 30 / 20 = 1.5 days under steady traffic and the same bad-event definition. That is a planning approximation, not a promise about a rolling window with changing request rates. Each paging alert should identify the affected objective, the team responsible for responding, a link to diagnostic information, and the first safe action to reduce the impact. Route slower budget erosion to a nonurgent work queue rather than paging on every symptom.

06Canary deployments, rollback, and backlog recovery

A canary release sends a limited portion of work to a new version before broad rollout. Compare workers running the new version with a control group running the current version on similar jobs. Measure whether photos become ready on time as well as whether the worker processes are running. In this example, route comparable jobs to a small canary worker pool with its own bounded queue so queue wait can be attributed to that pool. The canary shows elevated waiting and the reduced concurrency setting; stop expansion and restore the known-good configuration. If old and new workers instead pull from one shared queue, queue age is a shared symptom, not a per-version causal measurement. Compare per-version processing throughput and controlled workload evidence before attributing the delay.

Recover work accepted during the rollout

Keep data formats compatible

A schema change may prevent a simple binary rollback if the old code cannot read new data. Deploy changes in stages so that old and new application versions can both read the stored data during the transition. Feature flags can enable a new behavior separately from deploying the code. Limit the blast radius—the number of users or resources affected by one mistake—through gradual deployment and workload isolation. Keep a clear incident record of the symptom, change, action, and measured recovery.

Blue-green deployment: switch environments

A blue-green deployment prepares a second application environment, validates it, then shifts traffic from the old environment to the new one. It gives a clear traffic rollback target, but temporarily duplicates capacity and still needs connection draining: stop sending new work to the old environment while allowing its existing requests or connections to finish. A canary instead exposes a bounded cohort to the new version before broader rollout; either pattern needs comparable outcome measurements.

What traffic rollback cannot undo

07Disaster recovery: RPO, RTO, failover, and restore

A lost worker can be replaced and its jobs redelivered. A lost region may require a wider failover. A replicated bad deletion may require restoring older history. Choose the response from the actual failure, rather than treating every incident as a request to restart machines.

Recovery point objective, RPO, describes the acceptable loss of recent data measured in time. Recovery time objective, RTO, describes the target time to restore useful service. Both require tested procedures and measured results. For accepted photos, verify that the original files, records of pending processing jobs, publication status, and access permissions all survive recovery. Restoring one database does not by itself prove that users can upload and view photos again. Recovery guidance.

The multi-region-and-disaster-recovery chapter develops region placement and failback. Here the operational lesson is evidence: rehearse the recovery, check the customer-visible result, and record whether the objectives were met. A successful backup command or green failover control-plane status is only partial evidence.

Recovery planning also identifies the team responsible for recovery, a runbook with step-by-step instructions, accessible credentials and keys, and the dependencies needed to serve the recovered data. Verify that the remaining system can handle the required load when a server, zone, or region covered by the recovery plan is unavailable, and perform a restore to an isolated environment before relying on the procedure. Recovery point is a target: asynchronous replication lag must be measured to determine whether the observed lost work meets that target. Backups that share the same destructive permissions and retention policy as live data can fail together.

08Interview answer: explain how you know a service is healthy

Interviewer: “How will you know the upload service is healthy?”

Candidate: “I would measure both valid upload acceptance and whether accepted photos become ready within the agreed time. P501 returned success immediately but took 100 seconds, so an HTTP-success dashboard would miss the completion failure.

“I would trace acceptance, queue wait, rendering, and publication. The 95-second wait points toward processing capacity, and the canary’s reduced concurrency setting explains it. I would roll back that setting, verify the backlog drains, and check that replayed jobs preserve one correct result and private access. Alerts would focus on completion failures and error-budget burn.”

This answer connects monitoring to action: what the user needed, which measurements distinguish likely causes, what change is safe to undo, and how to check recovery. A monitoring box in a diagram needs those explanations.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.