Design a Multi-Tenant Vector Search Service

Interview problem: offer filtered semantic search over customer document passages. Support ingestion, updates, deletions, index upgrades and predictable search latency without leaking another tenant's records.

This design covers retrieval infrastructure. It does not include answer generation. Numbers are illustrative workload assumptions; approximate-search quality must be measured on the actual data and filters.

1. Requirements

Functional requirements

1. Upsert and delete versioned passages and their embeddings in a tenant namespace.
2. Search by a compatible query vector, required tenant scope and permitted metadata filters.
3. Return stable passage IDs, scores, source versions and pagination semantics where supported.
4. Rebuild or migrate an index without mixing incompatible embedding spaces.
5. Enforce quotas and offer stronger resource isolation for large tenants.
6. Support snapshots, recovery and auditable deletion status.

Non-functional requirements

1. Target p95 search latency below 150 ms for top-20 queries under the tested filter and concurrency distribution.
2. Target at least 95% ANN recall@20 relative to exact search on an agreed sampled workload; semantic relevance is a different metric.
3. Target 99.9% monthly search availability for admitted requests.
4. Agree a searchable-update bound, for example 60 seconds, and a stricter logical deny path for revoked/deleted data.
5. Never return records outside the authenticated tenant and current access scope.

2. Capacity estimates

Assume 100M passages, 768-dimensional float32 vectors and 2 KB of stored text/metadata per passage.

Dimension reduction and quantization change quality and resource use; neither guarantees faster filtered search. Capacity must include a concurrent rebuild and replica loss if those are required operating conditions.

3. Baseline

flowchart LR
 C[Client] --> A[Tenant-authenticated search API]
 A --> D[(Exact vector scan with metadata)]
 I[Versioned ingestion] --> D
 D --> A
 A --> C

Exact search is a useful correctness and ANN recall baseline for small collections and sampled evaluations. At 100M vectors, a full scan per interactive query may exceed the budget.

4. Evolve the design

Qdrant's multitenancy documentation describes payload-based, dedicated-shard and tiered approaches. These are deployment options; the API still has to derive and enforce tenant scope.

5. Detailed architecture

flowchart TD
 W[Authenticated writes] --> CAT[(Version and deletion catalog)]
 W --> LOG[Durable change log]
 LOG --> BUILD[Index workers and checkpoints]
 BUILD --> S1[(Shared small-tenant shards)]
 BUILD --> S2[(Dedicated large-tenant shards)]
 BUILD --> READY[Searchability checks]
 READY --> PUB[(Published index aliases)]
 Q[Authenticated query] --> POL[Current scope and deny checks]
 POL --> ROUTE[Placement and index-version router]
 PUB --> ROUTE
 ROUTE --> S1
 ROUTE --> S2
 S1 --> MERGE[Merge candidates and validate versions]
 S2 --> MERGE
 CAT --> MERGE
 MERGE --> R[Authorized results]
 S1 --> SNAP[Snapshots and recovery drills]
 S2 --> SNAP

Placement controls work distribution; authorization controls data disclosure. A shard key helps routing but is not a complete permission system.

6. APIs and records

PUT /collections/{id}/passages/{id} accepts a monotonic source version, embedding model revision, vector and metadata. POST /collections/{id}/search accepts the query representation and filters. DELETE creates a tombstone with a version; physical removal can proceed asynchronously after logical denial.

Define update acknowledgement: accepted into the log is not the same as searchable on all serving replicas. Return a status token when clients require read-after-publish confirmation.

7. Query and migration paths

1. Authenticate and derive tenant and resource scope in trusted code.
2. Pin a published index revision and placement generation.
3. Validate dimensions, embedding revision and supported filters.
4. Search eligible shards with bounded deadlines and candidate limits.
5. Merge and deduplicate, then reject stale/deleted/unauthorized records before release.
6. Return results plus the actual index revision and completeness status.

For an embedding migration, replay a snapshot and subsequent changes into a separate index, compare exact/ANN recall and product relevance, shadow representative filtered queries, then atomically change the alias. Keep rollback capacity without reintroducing revoked records; deletion state must dominate both versions.

8. Failure handling and operations

Test highly selective filters, dominant tenants, cold caches, concurrent ingestion and node loss. Average recall over unfiltered queries can conceal serious failures under real permissions.

9. Cost-benefit and closing

I would begin with exact-search truth sets and a shared filtered index, then isolate large tenants when measured skew justifies it. The design succeeds when updates, deletions, filters and migrations preserve the search contract—not merely when an unfiltered ANN benchmark is fast.

Q1: Is ANN recall the same as retrieval relevance?

Sample answer: No. ANN recall measures agreement with exact nearest neighbors under the selected vectors and distance. Those neighbors may still be irrelevant to the user's task. Evaluate semantic relevance and downstream evidence quality separately.

Q2: Can a one-minute TTL enforce immediate revocation?

Sample answer: No. It permits reuse until expiry. Use a current deny or policy-version check before disclosure, and keep physical cleanup separate from the access decision.

Q3: Why pin an index version during a query?

Sample answer: It keeps dimensions, model representation and placement coherent across fan-out. A request should not merge incomparable scores from old and new embedding spaces during a cutover.

Recall: Scope → Compatible representation → Filtered candidates → Current version → Explicit completeness.