DSPy: programming and evaluating model behavior

DSPy is a Python framework for composing language-model programs and optimizing parts of those programs against a chosen metric. You specify inputs, outputs, and program structure. An optimizer can search for better instructions or demonstrations using examples and evaluation feedback. Some optimizers support weight training; ordinary prompt optimization does not change model weights.

DSPy still creates prompts and makes model calls. Its value is a repeatable way to build and improve a program, not a guarantee of correct answers. This chapter focuses on implementation and release decisions. The prompt-optimization lesson develops the evaluation and search concepts in more detail.

Learn the six components

An optimizer's output is an application artifact: code/configuration plus selected instructions and examples, depending on the optimizer. It is not equivalent to a compiler proving program correctness. See the official signature guide and adapter guide.

Interview exercise: extract requirements from a design answer

Suppose a practice tool identifies requirements explicitly stated in a candidate's answer. It should distinguish “the system must support 10,000 requests per second” from a suggestion that this might be needed. The product must not silently invent a requirement and then grade the candidate against it.

Functional requirements

1. Extract stated functional requirements and non-functional constraints.
2. Return the exact supporting text for each extracted item.
3. Return empty lists when the answer does not state requirements.
4. Let the learner correct an extraction before using it in feedback.
5. Record the program and rubric versions that produced the result.

Non-functional requirements

1. Measure missed requirements and unsupported extractions separately.
2. Enforce an output schema and bounded processing time.
3. Keep private answers out of unrelated optimization datasets.
4. Compare performance by topic and answer length.
5. Retain a reproducible baseline and a rollback artifact.

Start with a single Predict module. Do not add a reasoning module, search optimizer, or repair loop until an observed error motivates it.

Define a precise interface

The following signature describes an extraction task. It assumes DSPy is installed; running a prediction additionally requires configuring an appropriate model. It is an interface example, not a complete grading service.

from typing import Literal
import dspy
from pydantic import BaseModel

class Requirement(BaseModel):
    kind: Literal["functional", "non_functional"]
    statement: str
    evidence_quote: str

class ExtractRequirements(dspy.Signature):
    """Extract only requirements explicitly stated in the answer.
    Quote the supporting text; do not add recommended requirements.
    """
    answer: str = dspy.InputField()
    requirements: list[Requirement] = dspy.OutputField()

extractor = dspy.Predict(ExtractRequirements)

Typed fields help describe and parse the expected output. Successful parsing is only the first check. A valid Requirement can still misclassify a suggestion or contain an invented quote. Independently check that each quote occurs in the supplied answer, then evaluate whether it supports the extracted meaning.

Use descriptive field names. evidence_quote is more informative than output_2. Avoid requesting a hidden internal reasoning transcript. If the product needs an explanation, request a brief evidence-based justification that can itself be reviewed.

Separate development from serving

flowchart TD
    A[Labeled development examples] --> B[Baseline program and metric]
    B --> C[Bounded optimization search]
    C --> D[Candidate artifact]
    D --> E[Held-out tests and error review]
    E --> F{Release criteria met?}
    F -->|No| G[Retain baseline and inspect failures]
    F -->|Yes| H[Versioned deployment]
    U[New learner answer] --> H
    H --> I[Prediction and independent validation]
    I --> J{Accepted?}
    J -->|Yes| K[Show editable extraction]
    J -->|No| L[Bounded repair or clear failure]

Optimization belongs in a controlled development process. A user request normally invokes the saved program; it should not trigger a fresh expensive search over the training set.

Choose an optimizer deliberately

MIPROv2 bootstraps demonstration candidates, proposes instructions, and searches combinations with Bayesian optimization. It can also optimize instructions without demonstrations. There is no universal “10–20 prompts” or “100–500 calls” budget; actual work depends on candidates, examples, trials, program depth, and settings.

Do not compare optimizers using their own best development scores alone. Give them a comparable budget and evaluate the selected artifacts on untouched examples. Review rare but costly failures separately from the average.

Design a metric that cannot win by doing nothing

For the extraction exercise, define the matching rules between a predicted requirement and an annotated requirement before scoring. Review ambiguous labels with more than one annotator.

An “all quotes are substrings” metric alone can be maximized by returning no requirements. An exact-text metric can unfairly reject valid normalization. Combine task-aware metrics with hard acceptance rules and inspected examples. Keep near-duplicate answers and variants of the same exercise in the same data split to reduce leakage.

A numerical example: suppose a search tests 40 candidates on 60 examples, with two model calls per program execution. That is 40 × 60 × 2 = 4,800 task-model calls before proposal, feedback, retries, and final evaluation. At an assumed average $0.001 per task call, that component costs $4.80. These are illustrative assumptions, not a DSPy quote or default.

Runtime refinement is not a hard guarantee

Older tutorials describe DSPy assertions as the current way to enforce constraints. The official legacy assertions page marks that mechanism deprecated and unsupported. Current code should use supported APIs and explicit application validation.

dspy.Refine runs a module up to a configured number of attempts, uses a reward function and feedback, and selects a result. Crucially, the selected result may be the best available prediction without meeting the desired threshold. Feedback generation can add calls beyond the number of module attempts. See the current Refine API and implementation.

For the extraction task:

1. Validate the response structure and evidence quotes.
2. If a repair is useful and the shared time/cost budget allows it, attempt a bounded repair.
3. Validate again outside the model's instructions.
4. If the result still fails, display a clear extraction failure or request manual correction.
5. Do not pass a rejected extraction into the grading stage as if it were trusted.

A prompt saying “do not reveal personal data” cannot establish a hard privacy property. Control what data is provided, where calls execute, and what outputs may be released. DSPy optimization and runtime refinement do not replace those controls.

Model changes and release management

A stable Python signature reduces some integration work when changing models. It does not ensure that every provider supports the same structured-output, tool-use, context, or generation behavior. An adapter may also use different formatting or fallback paths. Inspect the actual calls and failure modes.

Use this migration sequence:

1. Record the existing model, dependency versions, adapter, instructions, demonstrations, and metric version.
2. Run the current artifact on the proposed model without optimization.
3. Compare quality, cost, latency, and failure slices with the incumbent.
4. Optimize only if the expected improvement justifies the cost.
5. Run untouched release tests and a limited rollout.
6. Roll back on defined regressions and investigate them.

A new model release does not imply every prompt breaks. Recompilation does not automatically recover lost quality. Also inspect saved examples: an optimized prompt can embed training material, so the artifact needs the same data review as other application inputs.

Interview questions and answer notes

1. Does DSPy remove prompts? No. It provides program abstractions and methods to optimize how model calls are prompted.
2. Does a valid output type establish a correct extraction? No. Validate source support and task meaning separately.
3. Why can optimization improve validation scores but hurt production? Metric mismatch, repeated selection on a small validation set, leakage, or a traffic shift.
4. Does Refine(N=3) mean exactly three billable model calls? No. A module can make several calls, feedback adds work, and early stopping can reduce attempts.
5. What happens when no refinement meets the threshold? The application must verify the selected result and apply its own failure policy; selecting the best result is not acceptance.
6. Would you re-optimize immediately after changing providers? First evaluate the unchanged artifact and confirm integration behavior; then make a measured decision.
7. Why save more than the optimized instructions? Reproduction also depends on code, examples, model/configuration, dependencies, adapter, and evaluation versions.

Final notes

Remember signature → program → metric → search → independent evaluation → release. Start with a baseline, make the metric hard to exploit, and separate a higher score from a production guarantee. The strongest interview answer explains which failures are caught by ordinary code and which require empirical evaluation.

Next: Semantic Kernel and Microsoft Agent Framework.