Ensembles: use multiple outputs only when they improve the decision

An ensemble combines information from multiple predictors or model outputs to produce a decision. In LLM applications, this can mean voting over answers, selecting a candidate using scores, combining drafts or aggregating judgments. It is a technique to evaluate, not a requirement for every production system.

The potential benefit comes from useful complementary information. If every model repeats the same wrong assumption, adding more votes does not repair it. Different model names do not establish independent errors.

Distinguish the common patterns

Routing and fallback are related orchestration choices, but do not necessarily combine multiple predictions. See gateway routing. “Arbitration” describes choosing among candidates; it is not a universally separate category that forbids selection within an ensemble design.

A mixture of agents coordinates model calls at application level. A mixture-of-experts model routes computation among expert components inside a model. They are different architectures.

Interview scope: generate and review coding exercises

A learning service produces a solution and explanation for a programming exercise. Hidden tests and a human rubric establish whether it is useful. Model-written tests alone are insufficient: the same misunderstanding can influence both solution and tests.

Functional requirements

1. Generate candidates within a common task specification and allowed context.
2. Reject invalid or disallowed candidates before quality selection.
3. Run isolated executable checks where possible and evaluate explanations separately.
4. Return one selected answer, or abstain/escalate when no candidate qualifies.
5. Record candidate, scorer and selection versions for later comparison.

Non-functional requirements

1. Bound total generation, judging, retries, latency and spending per task.
2. Keep all providers within the task's access and data-processing constraints.
3. Prevent candidates from executing external writes while being compared.
4. Measure actual task success and error correlation on held-out examples.
5. Preserve a useful single-model baseline for quality and cost comparison.

Start with one generator and deterministic tests. Add more candidates only when the baseline's error analysis suggests a benefit that a stronger single model, better retrieval or a better prompt does not provide more efficiently.

Self-consistency: aggregate answers, not confidence claims

The original self-consistency research samples multiple solution paths and aggregates the resulting answers. For an arithmetic task, normalize final values and units before counting: 0.5 seconds and 500 ms should not become opposing votes. A unitless 500 may be ambiguous and should not be normalized by guessing.

For open-ended writing, exact string voting usually makes little sense. Semantic grouping is possible but introduces a grouping model or rules that can themselves make errors. Do not impose a fixed temperature or sample count as a universal recipe.

Worked probability: assume each of three binary classifiers is independently correct with probability 0.7 on a case. Majority correctness is:

$$P(\text{correct majority}) = 3(0.7)^2(0.3)+(0.7)^3 = 0.784.$$

Under the same assumptions, five classifiers reach 0.83692. These are calculations under an idealized model, not forecasts for a real LLM panel.

A 4/5 vote share is agreement, not an 80% probability that the answer is true. Calibrate any confidence estimate against labeled outcomes for the actual workflow.

Best-of-N: the selector matters as much as generation

Generate N candidates, apply hard eligibility checks, then rank candidates using a verifier or calibrated rubric. For code, use controlled tests and inspect security and maintainability separately. Passing an incomplete test suite does not prove a general specification.

Best-of-N can apply to mathematics as well as prose. A strong verifier may outperform simple majority voting. Conversely, a weak preference scorer can select fluent wrong answers from a better candidate set.

Optimizing harder against an imperfect score can worsen true quality. Reward-model overoptimization research studies this behavior, including best-of-N selection. Multiple reward models or conservative score aggregation can help in a measured setting, but neither guarantees immunity to reward hacking. Keep independent outcome checks and held-out tests.

Judge panels and pairwise evaluation

A panel grades the same output using a defined rubric. Normalize scales and handle missing/invalid judgments; do not silently average unrelated score meanings. Use median or trimmed means only when appropriate for the scale and sample count. Trimming the highest and lowest of two scores leaves no data.

The PoLL paper reports benefits from a diverse panel in its evaluated settings. It does not establish that every panel of small models beats every larger judge or that family labels alone measure diversity.

For a pairwise comparison, randomize answer order and, where the evaluation budget warrants it, compare both orders. Map choices back to candidate IDs before aggregation. If judgments disagree, the cause may be position bias, stochastic variation, ambiguity or a genuine tie; it is not automatically proven positional bias.

Do not calculate 1 - standard_deviation / mean and label it confidence. It can be negative, depends on score scale and has no general probabilistic interpretation. Track agreement separately from agreement with expert labels.

Debate and synthesis need external evidence

Critique can expose missing assumptions and tests. It can also propagate one model's error through the group. Keep initial answers independent before sharing them, require evidence or executable checks for disputed claims, and cap rounds by the task budget. There is no universal optimal two-round debate.

Mixture-of-agents research uses layered outputs as auxiliary information for later models. A production synthesis must preserve source provenance and validate the final answer; validating only the input drafts is insufficient. Compare that design against the simpler option of selecting an already valid candidate.

Evolve the baseline into a bounded ensemble

flowchart TD
    Q[Task, authorized evidence and shared budget] --> A[Independent candidate A]
    Q --> B[Independent candidate B]
    Q --> C[Independent candidate C]
    A --> V[Schema, policy and isolated task tests]
    B --> V
    C --> V
    V --> E[Score eligible candidates with versioned rubric]
    E --> S{Any candidate meets acceptance rule}
    S -->|No| H[Abstain, clarify or review]
    S -->|Yes| R[Select answer and validate final presentation]
    R --> U[Return one result]

If the chosen result proposes a real action, execution occurs later through a single authorized action path with a stable operation ID. Running three candidates must not create three tickets, payments or publications. Retries and cancellation must account for work already running or already billed.

Cost and latency: calculate the whole path

Illustrative prices per call:

• Four generators at $0.004 each: $0.016.
• Four candidate checks at $0.0005 each: $0.002.
• Final selection/presentation step at $0.001: $0.001.
• Total: $0.019 per task, or $950 for 50,000 tasks, before retries, infrastructure and review.

A single $0.004 generator costs $200 for the same volume before its checks. Compare cost per accepted outcome with the same acceptance standard; a higher raw quality score alone does not establish better economics. Apply the ensemble selectively if a measured routing rule identifies tasks that benefit.

Parallel execution reduces serial waiting, but waiting for all candidates still follows the slowest response. Approximately:

$$T_{\text{ensemble}} = \max_i(T_{\text{candidate},i}) + T_{\text{checks/selection}}.$$

If five independent calls each finish within three seconds with probability 0.95, all five do so with probability $0.95^5 \approx 77.4\%$. Shared congestion makes independence questionable and can worsen behavior. Fan-out also consumes more quota and concurrency, so parallelism does not guarantee single-call latency.

Run the comparison fairly

1. Freeze a representative held-out task set with important language, difficulty and risk slices.
2. Compare single model, stronger single model, repeated sampling, selected panel and any synthesis variant.
3. Keep evidence access, allowed tools and task budgets explicit.
4. Measure accepted outcomes, severe errors, abstention, human review, p95 latency and total cost.
5. Inspect paired wins/losses and uncertainty rather than reporting only average score differences.
6. Ship only the complexity supported by measured benefit; continue checking after model or traffic changes.

Interview tip: Explain which errors your extra candidate is expected to correct. “Three agents are more reliable” is not a mechanism.

Interview questions and answer checks

Final notes

Remember diversity → validation → aggregation → outcome measurement. An ensemble is useful when it corrects relevant errors at an acceptable cost. State the assumptions behind voting, validate the selector and final answer, and keep permissions and external effects outside the competition among candidates.