Model Selection and Optimization

Model Selection and Optimization is 16.8 percent of CCDV-F, roughly nine of 53 items, and the second-heaviest domain behind Applications and Integration. Those two together are half the paper, so study time should follow that shape: this domain earns deep work, while the one-item domains earn a skim.

It is tested as an engineering discipline, not a pricing quiz. Expect a workload described by its latency budget, volume, interactivity and reasoning depth, and a question about which tier and which optimization fit, or about why an optimization failed to pay off. Field-level detail counts: what invalidates a cached prefix, what a truncated response looks like in the API, how a thinking budget interacts with the output cap.

The idea that unlocks the domain is that cost and latency are not properties of the model you picked, they are properties of the tokens you send and the tokens you generate. Output tokens are billed at a multiple of input tokens and are produced one at a time, so they drive both the bill and the wall clock. Pick the cheapest tier that passes your eval, then reshape tokens, then measure.

• Start at the cheapest tier that passes your eval, then escalate. Choosing Opus first and optimizing later is the most common and most expensive mistake.
• Tier shorthand: Haiku for high-volume, latency-sensitive, well-specified work such as classification, extraction and routing. Sonnet as the balanced default for most production features. Opus for genuinely hard reasoning, long autonomous agent runs and ambiguous tasks.
• Cascade when most requests are easy: a cheap tier handles the bulk and escalates only the cases it flags as hard. Measure the escalation rate, because a 40 percent escalation rate erases the saving.
• Cost is input_tokens times the input rate plus output_tokens times the output rate. Output tokens cost several times more than input tokens, so a shorter answer format is a real optimization.
• Latency is dominated by output length, because tokens are generated sequentially. Input length mainly affects time to first token. Asking for a terse answer is usually a bigger win than a smaller model.
• Prompt caching matches an exact prefix. Put static content first and volatile content last, and mark the boundary with a cache_control breakpoint. Order is tools, then system, then messages.
• Caching pays for itself fast: a cache write costs a premium over normal input, a cache read costs a small fraction of it, so roughly the second hit within the lifetime is break-even. The default lifetime is short, on the order of minutes, and each hit refreshes it.
• Reuse of a long system prompt, tool definitions, a document, or a few-shot block across many requests is the signal to cache. One-off prompts and prompts below the minimum cacheable length are not.
• The Message Batches API trades latency for about half price, with results returned asynchronously within 24 hours. Use it for backfills, evals, bulk enrichment and offline scoring; never for a user-facing turn.
• Streaming does not reduce cost or total generation time — it reduces time to first token and lets you show progress. Use it for any interactive surface and for long generations that would otherwise sit past a request timeout.
• Extended thinking buys reasoning depth with output tokens. Set a thinking budget smaller than the output cap, keep it above the minimum, and raise it only when an eval shows the larger budget actually helps. Skip it for extraction, formatting and classification.
• Measure, do not assume. Read the usage block on every response, compare cost per task and p50 and p95 latency on a fixed dataset before and after, and confirm quality did not drop with the same eval.

• Cache miss from a moving prefix. A timestamp, session id, user name or retrieved snippet near the top of the system prompt changes the prefix every call, so nothing ever hits. Symptom: cache_creation_input_tokens on every request and cache_read_input_tokens near zero.
• Cache miss from tool churn. Reordering tools or editing one tool description invalidates the cached prefix and everything after it. Tool definitions sit before the system prompt in the cache order.
• Cache expiring between bursts. Traffic spaced wider than the cache lifetime pays the write premium every time and reads back nothing, so caching makes the workload more expensive, not less.
• Prompts below the minimum cacheable length are silently not cached. No error is raised; you only see it in the usage numbers.
• Silent truncation. When the output cap is reached the response stops mid-token-stream and stop_reason is max_tokens. Downstream JSON parsing then fails for a reason that looks like a model quality problem. Always branch on stop_reason, never infer it from a parse error.
• A thinking budget that starves the answer. The thinking budget is drawn from the same output allowance, so a large budget under a small cap leaves no room for the actual response.
• Dropping thinking blocks in an agent loop. When you feed prior turns back for tool use, preserve the thinking content returned to you; stripping it loses the reasoning the next turn depends on.
• Long non-streaming requests timing out. A large output cap on a single blocking request can exceed the request timeout; stream instead of raising the client timeout.
• Batch results are not guaranteed to come back in submission order and individual entries can fail independently. Correlate by the per-request identifier you set and handle partial failure.
• Runaway agent cost. Each turn resends the whole conversation, so token spend grows with the square of the turn count. Cap turns, prune or summarize history, and cache the stable prefix.
• Rate limits bite on tokens, not requests. Input and output tokens per minute are separate buckets and usually trip first; back off exponentially with jitter and honor the retry hint on a 429.
• Optimizing on one anecdote. A single fast response proves nothing. Without a baseline over a fixed set you cannot tell an optimization from noise, or notice the quality regression it caused.

STAMPS — the order to work through any cost or latency problem.

• S — Size the tier. Start with the cheapest tier that passes your eval and escalate only on measured failure. Consider a cascade before you consider a bigger model.
• T — Trim the tokens. Cut context going in, cut verbosity coming out, cap the output, and keep any thinking budget only as large as an eval justifies. Output tokens are the expensive ones.
• A — Anchor the prefix. Put static content first, volatile content last, and cache the boundary. An anchored prefix is what makes caching, and cheap repeat calls, possible.
• M — Move offline work to batch. Anything that can wait hours belongs in the Batches API at about half the price. Anything a user is watching does not.
• P — Perceived latency. Stream interactive responses so time to first token is short, and shorten the answer format, because generation time tracks output length.
• S — Score it. Re-run a fixed dataset, compare cost per task and p50 and p95 latency against the baseline, and confirm the quality eval did not move. An unmeasured optimization is a guess.