Model Selection Guide

Model selection is choosing a model and its operating configuration to meet a defined task, objective and constraints. In a production application, the decision includes the prompt, retrieval, tools, runtime and fallback—not only the model name. This lesson concerns deployment selection; statistical model selection during training is a related but different setting.

Reviewed September 24, 2026. The catalog is a dated shortlist using primary provider documentation. Prices are USD per million tokens at direct-provider Standard rates unless noted. Numerical interview workloads and candidate outcomes are illustrative. Use model taxonomy, capability assessment and pricing and costs for deeper reference.

flowchart TD
    T["Define task, correct outcome and constraints"] --> N{"Can a simpler non-generative system meet them?"}
    N -->|Yes| B["Keep it as a baseline candidate"]
    N -->|No| G["Screen model configurations against hard gates"]
    B --> G
    G --> E["Evaluate eligible complete workflows"]
    E --> P["Compare acceptable quality, latency and full cost"]
    P --> D["Choose the simplest justified configuration"]
    D --> C["Canary with rollback and reassessment criteria"]

Define the product decision

A model is one component of a system. A support assistant also depends on current policies, retrieval, permissions, tools, and a fallback when it cannot answer. If the wrong policy reaches the model, a more capable model may produce a more persuasive wrong answer. Selection therefore starts with the task and the failure, not with a leaderboard winner.

Write three example requests before comparing candidates: a straightforward policy lookup, a difficult question requiring several facts, and a consequential action such as changing a booking. Explain what a correct result looks like for each. A model may be adequate for lookup but unreliable at selecting authorized tool arguments. The capability requirement is attached to a workflow, not simply to the word “support.”

Separate the eligibility screen from the comparison. Eligibility includes data handling, permitted regions, license terms, available modalities, and capacity. Comparison includes measured quality, latency, cost, and maintainability among eligible options. An ineligible model does not become acceptable because it wins a weighted average.

Compare eligible configurations

Suppose candidate A costs $0.02 per task and resolves 92 of 100 representative tasks, while candidate B costs $0.01 and resolves 88. These invented figures are not enough to choose. Inspect the failures. If A makes unauthorized changes while B safely hands off, their averages conceal a crucial difference. If B's additional handoffs require expensive human time, its lower model price may not reduce total cost.

Next compare latency at expected concurrency, not just one developer's sequential calls. Include long requests, output length, rate-limit behavior, and retries. A provider can accept the model ID yet lack sufficient quota for your launch. “Supports a million tokens” likewise says what input may be accepted, not whether the model reliably uses every fact or whether that request meets the response deadline.

Now decide whether routing is justified. Routing selects where a request or step executes. Pre-inference routing uses information available before generating the answer. A cascade tries one model and escalates based on a validated result check; it is one way to route a workflow. Both add components that can make mistakes. Model-generated confidence is not automatically a safe escalation signal. A schema validator, verified task result, or calibrated risk classifier may provide better evidence, depending on the task.

Finally document the choice, runner-up, assumptions, and reassessment trigger. Keep prompts, tool schemas, and evaluation cases versioned together. An abstraction layer can standardize logging and errors, but it does not make reasoning controls, caching, tool behavior, or data contracts identical across providers. Those differences need adapter tests and workload evaluation.

Decision sequence

Memorize the decision sequence: task, gates, evidence, economics, rollout. Look up the current model IDs, rate cards, quotas, and contractual terms when making the decision. The detailed catalog below is dated reference material; it is not a permanent ranking or a set of promises about your workload.

Table of Contents

• The Core Principle
• Selection Framework
• Current Model Landscape
• Use Case Mapping
• Evaluation and Rollout
• Cost Analysis
• Operational Considerations
• Multi-Model Strategies
• API vs Open Weights
• Selection design interview
• Interview questions
• Official Sources

The Core Principle

There is no universally best model. There is only the best validated choice for a specific workload, traffic shape, risk level, and operating environment.

Provider descriptions and public benchmarks are useful for building a shortlist. They are not a substitute for testing the exact prompt, tools, documents, languages, output schema, and failure modes used by the product.

When the product objective is to minimize cost subject to fixed requirements, use this constrained objective:

$$\text{Choose the lowest-total-cost candidate that satisfies the quality, safety, latency, capacity, and governance requirements.}$$

The strongest model may be appropriate for a difficult agent step and wasteful for classification. The cheapest model may have the lowest token bill but the highest cost per successful task because it needs retries or human correction.

Selection Framework

Step 1: Define hard gates

Eliminate candidates that cannot meet non-negotiable requirements:

A large context window is only a capacity limit. It does not prove accurate recall, reasoning, or citation at that length. Test long-context behavior at the actual document mix and token distribution.

Step 2: Build candidates by operating lane

Choose a small shortlist from the lanes relevant to the task, including the existing or simplest viable baseline. A basic classifier does not need a candidate from every lane:

These are candidates, not cross-provider rankings. For example, OpenAI calls GPT-6 Astra its latest flagship, while Anthropic offers Claude Fable 5.1 for demanding reasoning and long-running agents. Only a workload-specific evaluation can resolve that choice.

Step 3: Score the finalists

Use pass/fail gates before weighted scoring. A model that violates a residency requirement or fails a safety threshold cannot compensate with a better coding score.

Pareto dominance means one eligible option is no worse on every chosen objective and better on at least one. A Pareto frontier contains options not dominated this way. Avoid inventing one global score too early: weighted scores depend on units, normalization and stakeholder preferences. Keep hard gates visible and show the trade-off frontier: the fastest passing model, the cheapest passing model, and the highest-quality model.

Current Model Landscape

General-purpose API shortlist

The table gives shortlist candidates, not recommended winners. “Context” reports each provider’s published window/input convention; it is not a universal maximum-prompt measure. Input/output prices omit cache, tools, service, regional and contractual modifiers. Current pricing detail.

Important constraints: GPT-6 models reserve distinct maximum input/output limits; all three allow at most 128K output. Prompts above 272K use higher whole-request rates. Claude Opus 5.5 cache reads are 5% of input, Fable 5.1 2.5%, Sonnet/Haiku 10%. Gemini 3.8 Flash’s announced January 2027 price is higher. Grok’s higher rate starts at 200K prompt tokens. These are reasons to keep the detailed, versioned rate card beside the decision.

DeepSeek is also eligible for shortlisting where its contract fits: deepseek-flash currently serves V4.1 Flash, with peak USD .30 input miss / .006 cache hit / 1.20 output; deepseek-v4-pro serves V4 Pro 0813 at 1.32 / .044 / 3.96. Off-peak rates are half. Both publish a 1M window and 384K maximum output; Flash supports image input while Pro does not. Legacy Flash aliases route to V4.1 Flash. DeepSeek model and pricing contract.

Preview and limited-access models

Keep these out of a default production baseline unless the release risk is explicitly accepted:

A provider offering an endpoint does not establish that your account has approved access, quota or contractual permission. Confirm these before planning production traffic.

Migration checks for the newest models

Sources: GPT-6 Sol, GPT-6 Luna, Astra guide, Opus 5.5, Gemini 3.8 Flash, Grok 4.7.

For new video work, do not choose Sora 2: the Videos API and Sora 2 models shut down September 24, 2026. OpenAI self-serve fine-tuning is also winding down; do not assume a new organization can train hosted custom models. OpenAI lifecycle notices.

What the table does not prove

It does not prove which model is best at coding, science, legal reasoning, long-context recall, safety, or agent autonomy. Provider-written descriptions are first-party positioning. Benchmark scores also depend on prompting, tool access, reasoning settings, sampling, grading, contamination, and harness implementation.

Use public results to select candidates. Use representative, versioned evaluations with checked grading and stated uncertainty to select a production configuration; private tests can also be biased.

Use Case Mapping

Two common mistakes:

1. Using context size as the only RAG criterion. Retrieval can reduce latency, cost, distraction, and access-control risk even when the entire corpus technically fits in context.
2. Using a general model for a specialist workload. A text model that can reason about an audio transcript is not necessarily the right transcription engine; a vision-language model is not necessarily the right OCR system.

Evaluation and Rollout

Build an evaluation set from real traffic

Use representative examples plus a separately reported adversarial/risk suite. Targeted cases do not automatically have their production frequency. A useful initial set contains:

• common requests weighted by real frequency;
• rare but high-impact requests;
• multilingual and formatting edge cases;
• long inputs near the actual P95 and maximum;
• malformed inputs and prompt-injection attempts;
• every tool and output schema;
• tasks that the current system gets wrong;
• abstention cases where the correct behavior is to refuse or escalate.

Keep the evaluation set versioned. Separate a development set used for prompt tuning from a held-out set used for model selection.

Evaluate the complete system

For an agent, the unit under test is the full trajectory:

$$\text{request} \rightarrow \text{model decision} \rightarrow \text{tool calls} \rightarrow \text{state changes} \rightarrow \text{final result}$$

A model can have excellent single-turn benchmark results and still fail because it chooses the wrong tool, repeats an irreversible action, loses state, or cannot recover from a tool error.

Record at least:

• final-task success and grader rationale;
• every model, prompt version, and reasoning setting;
• input, cached-input, reasoning, and output token counts;
• tool count, tool errors, and retry count;
• time to first token and end-to-end latency;
• provider/model error code and fallback behavior;
• estimated and invoiced cost.

Roll out safely

1. Offline replay: compare finalists on the held-out evaluation set.
2. Shadow traffic: send production-shaped requests without exposing the new output or allowing side effects.
3. Canary: expose a small, monitored cohort with explicit rollback criteria.
4. Ramp: increase traffic only while quality, safety, latency, and cost stay within limits.
5. Continuous evaluation: rerun the suite on every model, prompt, tool, or routing-policy change.

Pin a dated model version when the provider offers one and reproducibility is important. If only a rolling ID is available, treat provider-side changes as unannounced migrations: monitor the model identifier returned by the API and keep a rollback path. The returned name may remain unchanged even when behavior changes, so watch task-level regressions as well as identifiers.

Cost Analysis

Model the whole request

Sum cost over every execution attempt, with provider-specific non-overlapping input, cache-write, cache-read and billed-output quantities. Add distinct tool/media charges once and storage over its billing interval. Include failed work, fallback, shadow and evaluation traffic, then application operations and human review. A monthly shared storage bill is not a new fee on each request. See the executable cost calculation.

The decision metric should usually be:

$$\text{Effective cost per success} = \frac{\text{total system cost}}{\text{successfully completed tasks}}$$

Worked example

Assume 1 million requests per month, each with 1,000 uncached input tokens and 500 total billed output tokens, including any billed reasoning tokens. That is 1,000 million input tokens and 500 million output tokens. Actual reasoning runs may use much more output. This illustration uses current standard rates and excludes caching, tools, retries, long-context tiers, and discounts:

This table is not a recommendation. If the USD 450 model completes half of one million tasks, its token-only cost is USD .0009 per success. At 95% completion, the USD 2,625 model costs about USD .002763 per success. The first is still cheaper on that narrow metric, but it fails a 95% completion requirement. Quantify the additional repair/fallback costs before comparing acceptable complete systems.

Astra's standard input/output rates are 5× GPT-6 Sol's. That does not prove 5× the cost per completed job: token consumption, retries, and success rate can change. Measure them rather than assuming the newer model is either cheaper or automatically worth the premium.

Cost controls that preserve quality

• Put stable instructions and reusable documents first to improve prefix-cache reuse; measure actual cache hits.
• Cap output length and reasoning effort by task rather than using one global maximum.
• Use batch or deferred processing for work without interactive latency needs.
• Route only when routing accuracy and added latency have been measured.
• Retrieve the smallest relevant context instead of sending an entire corpus.
• Detect loops with per-request token, tool-call, wall-clock, and dollar budgets.
• Track cost by tenant, feature, task type, route, and success outcome.

Operational Considerations

Rate limits are account-specific

Do not copy a static RPM/TPM table into an architecture document. Limits vary by model, usage tier, account, region, and processing mode, and they change. Read the active limits from the provider console or API, load-test below the approved quota, and maintain headroom for retries and bursts.

Plan separately for:

• requests per minute and tokens per minute;
• concurrent requests or sessions;
• batch queue limits;
• long-running streams and realtime connections;
• tool-specific and regional quotas;
• quota propagation time after an increase.

A common interface is not identical behavior

An abstraction layer is useful for routing and observability, but it should not pretend that providers have the same features. Maintain a capability contract for each exact deployment configuration:

Use supported, unsupported and unverified states rather than assuming every provider has every feature. An unverified required feature blocks that route until checked. A basic JSON-output feature does not establish strict schema enforcement, and two individually supported features may be incompatible together.

The executable example below demonstrates request rejection. Provider adapters additionally need contract tests for roles, tool-call representation, streaming events, usage fields, image formats, errors, timeouts and cancellation.

Reliability and fallback

Use bounded exponential backoff with jitter for eligible transient throttling and 5xx errors. First classify whether a request may already have executed; do not blindly repeat an action after an unknown outcome. Respect retry headers and overall deadlines. Do not retry invalid requests, safety refusals, or non-idempotent side effects blindly.

Cross-provider fallback is a new inference path, not merely a second URL. It may need:

• a provider-specific prompt and tool schema;
• conversion of conversation and tool state;
• revalidation of structured output;
• a separate safety policy;
• confirmation that data may be sent to the fallback region/provider;
• a remaining latency and dollar budget.

Use circuit breakers and failure-class-aware routing. Record when a fallback occurs so degraded service does not silently become the normal service.

Governance and lifecycle

Before production approval, document:

• data retention and training-use terms for the exact paid/free tier;
• deployment region and subprocessors;
• model ID, alias behavior, deprecation notice, and rollback target;
• safety controls and human escalation path;
• output ownership and media-rights requirements;
• audit logging, access control, and incident response;
• whether provider search, files, containers, or other tools retain data under different rules from the base model call.

Multi-Model Strategies

1. Evaluated routing

Route by requirements that are observable before inference: modality, context size, latency class, tenant policy, and task type. A learned complexity router can be added only after its routing errors are measured.

2. Cascade

Try a lower-cost model first, accept only when a calibrated validator says the result meets the task contract, and escalate otherwise. The validator must be tested for false acceptance; “the model sounds confident” is not a gate.

3. Specialist routing

Use dedicated transcription, OCR, embedding, moderation, realtime, image, or video endpoints for the portions they are designed to solve. A workflow can use several specialists and one general reasoning model.

4. Provider fallback

Fail over on a narrow set of transient failures. Keep provider-specific prompts and tools tested continuously so the backup path is not discovered to be broken during an outage.

5. Draft and verify

Use one model to draft and another model, deterministic checker, or human to verify high-impact work. Correlated errors matter: two calls to the same model can repeat an error, and different providers can also fail together on the same misleading evidence. Measure the verifier’s false-acceptance rate.

6. Shadow and canary

Use shadow traffic to compare a new model without user impact, then a canary to measure real-world outcomes. This is safer than changing a rolling alias for all traffic at once.

Multi-model systems add routing errors, extra latency, larger failure surfaces, more vendor contracts, and more observability work. Add them only when measured quality, resilience, or cost gains exceed that complexity.

API vs Open Weights

There is no universal request-volume crossover point. Self-hosting economics depend on model size, quantization, accelerator type, utilization, batching, latency target, redundancy, region, staffing, and the quality loss relative to the API candidate.

Prefer a managed API when

• fastest time to market and immediate access to current models matter;
• traffic is bursty or difficult to capacity-plan;
• the team does not want to operate GPU scheduling and model serving;
• managed tools, safety systems, and enterprise support are valuable;
• workload data is permitted under the provider contract and deployment terms.

Prefer open weights or controlled hosting when

• weights or inference must remain in a controlled environment;
• the workload needs weight-level customization or a specialized serving stack;
• traffic is predictable enough to keep accelerators highly utilized;
• the team can own security patches, serving, evaluation, upgrades, and availability;
• the chosen license permits the intended use.

Compare total cost of ownership

Include accelerators, idle and failover capacity, networking, storage, orchestration, observability, engineering/on-call time, security, model evaluation, upgrades, and the business cost of lower task quality. Benchmark with the intended precision, context length, batch size, and concurrency; a single throughput number from a different serving setup is not a capacity plan.

The current Mistral lineup offers a useful range for this evaluation: Large 3 and Small 4 use Apache 2.0 weights, while Medium 3.5 uses a Modified MIT license. Read the exact license and model card before deployment.

Manager interview practice

Decision example: two candidates pass ordinary quality tests, but one fails a required tool-permission case. Eliminate that configuration until fixed; a better mean score cannot compensate for a hard requirement. For the remaining candidate, test peak capacity and actual rework before committing.

Recall checks: Which assumptions would reverse your choice? What is the fallback's evidence? Which version, prompt, tool, and data configuration did you compare? What would justify a second provider or self-hosting?

Assign an owner to reassess on model retirement, a price change, a new region or data requirement, or a material workload shift. Memorize the method; look up the current catalog.

Executable adapter contract example

A common application interface should preserve material differences between providers. This standard-library example uses fake, already-reviewed capability contracts to show rejection before execution. Unverified contracts must not reach this constructor as approved booleans. It makes no claim about any provider's current SDK.

from dataclasses import dataclass

@dataclass(frozen=True)
class Capabilities:
    structured_output: bool
    tools: bool
    max_output_tokens: int

    def __post_init__(self):
        if type(self.structured_output) is not bool or type(self.tools) is not bool:
            raise ValueError("capability flags must be booleans")
        if type(self.max_output_tokens) is not int or self.max_output_tokens <= 0:
            raise ValueError("capability output limit must be a positive integer")

class DemoAdapter:
    def __init__(self, name, capabilities):
        if not isinstance(name, str) or not name or not isinstance(capabilities, Capabilities):
            raise ValueError("a named capability contract is required")
        self.name, self.capabilities = name, capabilities

    def generate(self, *, needs_schema, needs_tools, max_output_tokens):
        if type(needs_schema) is not bool or type(needs_tools) is not bool:
            raise ValueError("request flags must be booleans")
        if type(max_output_tokens) is not int:
            raise ValueError("output budget must be an integer")
        c = self.capabilities
        if needs_schema and not c.structured_output:
            raise ValueError("Structured output is a required capability")
        if needs_tools and not c.tools:
            raise ValueError("Tool use is a required capability")
        if not 0 < max_output_tokens <= c.max_output_tokens:
            raise ValueError("Unsupported output budget")
        return {"provider": self.name, "status": "demo_only",
                "output": None, "usage": None}

adapter = DemoAdapter("candidate-A", Capabilities(True, False, 2048))
assert adapter.generate(needs_schema=True, needs_tools=False,
                        max_output_tokens=256)["status"] == "demo_only"

In a real adapter, preserve finish reasons, truncation, refusal, tool-call IDs, cancellation behavior, and usage categories. Do not turn unsupported tool use into plain text and pretend the same contract succeeded. Run the same application acceptance cases against each concrete adapter, plus provider-specific cases for features whose semantics differ. Fallback eligibility depends on these contracts as well as availability.

Interview: select and deploy an invoice-extraction system

Problem: choose a configuration that extracts invoice fields for a reviewer. The system prepares a draft record; it does not pay invoices or change supplier bank details. All figures below are interview assumptions.

Requirements and evaluation plan

Functional requirements

1. Accept authorized invoice uploads and extract supplier, invoice number, date, currency, subtotal, tax and total.
2. Attach document/page evidence to each extracted field; distinguish absent, ambiguous and unreadable fields.
3. Validate types, arithmetic and document identity outside the model, then present a reviewable draft.
4. Escalate unsupported layouts and uncertain results while preserving the original document.
5. Record the exact model/prompt/OCR/validator configuration and corrections for reassessment.

Non-functional requirements

1. All document processing, tools, storage, logs and fallbacks must use the approved EU processing path.
2. A prespecified two-sided 95% Wilson lower bound for complete-draft correctness before human review must exceed 95% on the representative holdout; evaluate critical errors separately.
3. Automated draft-generation p95, including queueing and validation, must stay below eight seconds at a five-document/second peak for the defined one-page interactive workload. Measure human review/approval time separately.
4. Input is bounded to 20,000 model tokens and 1,200 output tokens per interactive attempt; larger jobs use a separately evaluated asynchronous route.
5. Authorization, field provenance and duplicate-upload handling must hold under retries and worker failures; no autonomous payment actions are permitted.

Use 1,000 independent representative held-out documents, separating related vendor templates appropriately during development/test splitting. Add a separately reported permission, misleading-content and critical-field suite. Compare complete extraction workflows with the same authoritative references, validators and budgets. A deterministic OCR-plus-rules baseline remains in the comparison.

A’s 95% Wilson interval is approximately 95.06%–97.39%, narrowly passing the stated quality rule. That is evidence under the sampling assumptions, not protection against dataset bias. Do not claim A is statistically superior to B: their marginal counts alone omit paired fixes/regressions. B’s deadline failure is enough to exclude it from this interactive route. None of the scores establishes that permission leaks are impossible.

Start simple, find the failures, then improve

Initial design: upload → OCR → one model call → JSON → reviewer. It provides a fast prototype. JSON can be valid but factually wrong; uploads can be repeated; a model alias can change; a fallback can send private invoices outside the approved region; an interrupted call can leave its result unknown.

Detailed design

flowchart TB
    U["Authenticated upload + document limits"] --> O[("Tenant-scoped original and digest")]
    O --> Q["Idempotent job + bounded queue"]
    R["Versioned holdout, operating tests and decision"] --> C[("Approved configuration registry")]
    C --> W["Worker: approved OCR + model adapter"]
    Q --> W
    W --> V["Schema, arithmetic, identity and evidence checks"]
    V -->|Valid draft| D[("Versioned draft record")]
    V -->|Uncertain or failed check| H["Reviewer / documented exception route"]
    D --> H
    H --> A["Explicitly approved record; no payment execution"]
    W -->|Provider outcome unknown| X["Reconcile attempt; bounded permitted retry"]
    X --> V
    D --> E["Outcome samples, corrections, latency and full cost"]
    A --> E
    E --> R

The registry contains the approved model, endpoint, region, prompt, OCR version, validator and fallback policy. A successful model call does not itself approve a business record. Every field links to the relevant document revision; a correction creates a new draft revision. A changed document or configuration requires revalidation.

API/state contract: POST /extractions accepts a client idempotency key under the authenticated tenant and binds it to the uploaded digest and extraction configuration. Reusing a key with different input returns a conflict. A job moves through queued → running → draft-ready or needs-review/failed. Persist attempt IDs and leases; an unknown external result is reconciled before another attempt is dispatched. A reviewer updates a specific draft revision, so a late worker cannot overwrite the correction. Idempotency in our store prevents duplicate drafts; it does not guarantee a provider executed or billed only once.

Capacity: at five documents/second and an illustrative mean active processing time of two seconds, Little’s Law gives ten active jobs on average. Provision and test bounded concurrency—for example, a candidate limit of twenty—with headroom for tails, provider quotas and one worker loss. That limit alone does not prove the eight-second p95. At the 20,000-token input bound, five calls/second could demand six million input tokens/minute before retries. Check both token and request quotas against the actual input distribution.

Full economics and decision

For 50,000 one-page documents/month, use these invented prices and operating assumptions. Exception review means additional correction work. Mandatory final approval is budgeted separately for every document at an illustrative fifteen seconds each; validate that workload too.

Estimated saving is USD 9,300/month. If both processes produce 48,000 verified accepted records after review, costs are about USD 608.33 and USD 414.58 per 1,000 accepted records. Validate the four-percent review rate and accepted-outcome assumption in the canary; field accuracy alone does not establish reviewer workload.

The proposed lines excluding exception review total USD 15,900. Its TCO reaches the USD 29,200 baseline at 6,650 reviewed documents, a 13.3% review rate. Above that rate, the proposed cost advantage disappears under these assumptions. Additional proposal-specific capacity, retries or approval labor would lower that break-even rate; equal new costs on both sides cancel in this comparison. Do not transfer this result to another product without recalculating.

Rollout: approve the exact A configuration after the holdout, critical-error and load gates pass. Shadow only within the approved data path, then canary by stable tenant/document cohort. Revert on a reproduced permission violation, corrupted critical field, material deadline breach or failed agreed economic guardrail. Keep the original documents, old draft revisions and a tested reviewer path. Analyze statistical improvement at its planned horizon; a safety stop is not a claim of superiority.

Closing remarks: “A fits the interactive requirements; B is worth considering separately for asynchronous work, and C is excluded by the data contract. I would keep the initial deployment to one model configuration and a reviewer path. The estimated saving is mostly reduced exception work, so the canary must verify that assumption. I would add a router only if its measured benefit exceeds the extra errors, state handling and operating cost.”

Interview questions with developed answers

Q1: How do you choose between models from OpenAI, Anthropic, and Google for production?

Q2: When would you self-host rather than use an API?

Q3: When is a multi-model router worth building?

Q4: A model wins a benchmark but loses your evaluation. Which do you trust?

Q5: How do you prepare for provider model updates?

Q6: A candidate leads quality and cost but uses an unapproved region. Can a weighted score select it?

Q7: Does a 1.05-million-token window accept a 1-million-token prompt?

Q8: The provider supports tools and reasoning. Why does the migrated call fail?

Q9: Configuration A has 964 correct drafts and B has 971. Which is better?

Q10: Can a model grader make a cheap-first cascade reliable?

Q11: Why retain a deterministic baseline during model selection?

Q12: A fallback is healthy but has no validated tool-state conversion. Should you use it during an outage?

Q13: The model alias has not changed. Is another regression run unnecessary?

Q14: The token bill increases by USD 500. Why might the new system still be worthwhile?

Q15: When should a team add a second production model?

Final recall table and notes

Tip: make the decision explicit. “A is the current choice because it passes these requirements at this total cost; B is excluded by latency; C is excluded by data policy” is more useful than ending with a list of vendor names. State assumptions that could change the answer. A new model release is a reason to evaluate, not automatic evidence to migrate.

60-second interview answer

I choose a production configuration by hard requirements first, then workload evidence. I shortlist models that meet data, modality, lifecycle, and capacity constraints, compare quality and serious failures on the same representative tasks, and measure complete-task latency and cost. I choose the simplest passing option, document the tradeoff, and canary it with rollback. A public benchmark or provider description helps identify candidates; it does not establish the best choice for our application.

Remember: Gates → workload tests → total cost → controlled rollout.

Official Sources

Model contracts and rate cards checked September 24, 2026:

• OpenAI model catalog and API pricing
• GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna
• Anthropic current-model overview and Claude API pricing
• Google Gemini model catalog and Gemini Developer API pricing
• xAI models and API pricing
• Mistral model catalog, pricing, and known platform limitations
• DeepSeek models and pricing

Next: Fine-Tuning Guide