Cut AI Costs: Model Routing and Batching with Claude

By MUASIF80·June 5, 2026·25 min read·AI Use Cases

Why Claude Cost Optimization Matters Before You Scale

At prototype scale, token costs feel irrelevant. You run a few hundred requests a day, the bill is $10, and no one notices. The trap springs when you hit production. A SaaS product that handles 100,000 daily AI requests at $0.015 per thousand output tokens from Opus runs you $4,500 a month in LLM costs alone. The same traffic routed intelligently can cost under $400.

The math is not theoretical. Anthropic publishes list prices, and the gap between model tiers is enormous. Most production workloads contain a mix of tasks: some genuinely hard (multi-step reasoning, long document synthesis, code generation requiring deep context), and some trivially easy (intent classification, sentiment labeling, single-sentence extraction, yes/no judgements). Sending all of them through the same heavy model is the equivalent of using a freight truck to deliver letters.

Model routing solves this. You add a thin decision layer that classifies each incoming request, assigns it to the cheapest model capable of handling it correctly, and lets heavier models stay reserved for cases that genuinely need them. Paired with batching for non-real-time workloads, you can cut your LLM spend by 60 to 80 percent without touching output quality.

This article builds the complete system: a classifier that runs on Haiku, a router that dispatches to Haiku, Sonnet, or Opus based on the classification result, and a batch processor for bulk jobs. All code is real, runnable Python against the current Anthropic SDK.

The Model Tier Landscape: What Each Model Is Good At

Haiku: High Volume, Low Complexity

Claude Haiku (claude-haiku-4-5) is the workhorse for classification, routing decisions, short extractions, yes/no checks, and any task where the prompt is simple and the answer is a single label or short phrase. It is the fastest model in the Claude family and the cheapest. Think of it as the model that should handle the majority of requests in any well-designed production system.

Good fits: intent detection, spam filtering, sentiment scoring, extracting a single field from a structured document, language detection, simple Q&A over a short context, routing decisions themselves.

Sonnet: The Production Default

Claude Sonnet (claude-sonnet-4-6) is where most substantive work should land. It handles multi-step reasoning, code generation, summarization of moderate-length documents, and RAG response synthesis. The cost is significantly higher than Haiku but a fraction of Opus. For the majority of “real” tasks in a production app, Sonnet gives you the quality you need at a price that scales.

Good fits: code review, pull request summaries, contract clause extraction, customer support responses, structured JSON generation, SQL generation, moderate-length summarization.

Opus: Reserved for the Hard Cases

Claude Opus (claude-opus-4-8) is the most capable model and the most expensive. It is the right choice for tasks requiring deep multi-step reasoning, long-form creative work, expert-level analysis across a large context window, or anything where accuracy failure has high business cost. In a well-tuned routing system, Opus should handle 5 to 15 percent of traffic, not 100 percent.

Good fits: complex legal or financial analysis, multi-document reasoning, long-form technical writing, advanced code generation with architectural constraints, tasks where you have measured that Haiku and Sonnet produce unacceptable failure rates.

How the Router Works

The Classification Step

The router makes a cheap Haiku call first. It sends the user’s request text to a concise system prompt that asks for a complexity label: simple, moderate, or complex. The classification call uses a small max_tokens budget (32 tokens is enough for a one-word answer). Total cost for this step is a few cents per thousand requests.

The classifier prompt defines what each label means. Getting this definition right is where you spend the most engineering time. A good classifier system prompt includes 2 to 3 examples per label, a clear definition of what distinguishes moderate from complex, and a fallback rule (“when in doubt, return moderate”). You tune it by sampling 200 to 300 real requests from your logs, labeling them manually, and measuring classifier accuracy.

Routing Logic

Once you have the label, the router picks a model. The mapping is configurable. A simple default:

• simple: Haiku. Single-step tasks, short outputs, classification or extraction jobs.
• moderate: Sonnet. Multi-step reasoning, code, summaries, anything requiring nuance but not rare expertise.
• complex: Opus. Long-context analysis, tasks where previous Sonnet runs showed quality degradation, anything business-critical with no tolerance for error.

You can add override rules. For example: always use Sonnet or above for customer-facing responses regardless of complexity label, or force Haiku for all background batch jobs irrespective of classification to cap costs during a high-volume processing window.

Batching for Non-Real-Time Work

Some workloads do not need an immediate response. Report generation that runs overnight, bulk content enrichment, nightly data labeling jobs: these can be collected into batches and submitted together. Benefits include reduced per-call overhead, the ability to schedule off-peak to avoid rate limit pressure, and potential eligibility for batch pricing discounts where Anthropic offers them.

The batching pattern in this article is straightforward: requests accumulate in a queue (in-memory for the POC, Redis or SQS in production). A background coroutine drains the queue in configurable chunks (e.g., 50 items per batch). Each chunk is processed concurrently with asyncio, respecting a concurrency limit to avoid hitting rate limits. The batch processor logs token usage per job so you can account costs back to the originating service or customer.

Cost and Latency Trade-offs

Model routing introduces one new latency cost: the classifier call. On Haiku, a 200-token classification request typically returns in 200 to 600 milliseconds. For interactive use cases where total latency must stay under 1 second, this is a material overhead. For anything tolerating 2-plus seconds (most document processing, most backend pipelines), it is negligible.

Two mitigations help with latency-sensitive paths. First, cache classifier results for identical or near-identical inputs using a hash of the request text. If the same request recurs (common in classification-heavy apps), you pay the classification cost once. Second, run the classifier and the first-tier model call in parallel. If the classifier result comes back before the speculative Haiku call completes, you win with no latency penalty. If the classifier says “complex” and the speculative call was on Haiku, you cancel it and re-issue on Sonnet or Opus. This is the speculative execution pattern, appropriate for latency-critical APIs.

The Complete POC: Installation and Setup

Requirements

# requirements.txt
anthropic>=0.30.0
python-dotenv>=1.0.0

# .env
ANTHROPIC_API_KEY=sk-ant-your-key-here

Full Source Code

"""
Part 27 POC: Claude Cost Optimization via Model Routing and Batching
Save as: router.py
Run: python router.py
"""

import asyncio
import os
import time
import hashlib
from dataclasses import dataclass, field
from typing import Optional
from dotenv import load_dotenv
import anthropic

load_dotenv()

# ---------------------------------------------------------------------------
# Model configuration
# ---------------------------------------------------------------------------

MODEL_HAIKU = "claude-haiku-4-5"
MODEL_SONNET = "claude-sonnet-4-6"
MODEL_OPUS = "claude-opus-4-8"

# Approximate public pricing per 1M tokens (update from anthropic.com/pricing)
PRICING = {
    MODEL_HAIKU:  {"input": 0.80,  "output": 4.00},
    MODEL_SONNET: {"input": 3.00,  "output": 15.00},
    MODEL_OPUS:   {"input": 15.00, "output": 75.00},
}


# ---------------------------------------------------------------------------
# Data structures
# ---------------------------------------------------------------------------

@dataclass
class RoutedRequest:
    task_id: str
    user_prompt: str
    system_prompt: str = "You are a helpful assistant."
    max_tokens: int = 1024
    # Optional override: force a specific model regardless of classification
    force_model: Optional[str] = None


@dataclass
class RoutedResponse:
    task_id: str
    model_used: str
    complexity_label: str
    response_text: str
    input_tokens: int
    output_tokens: int
    classifier_input_tokens: int
    classifier_output_tokens: int
    latency_seconds: float

    @property
    def cost_usd(self) -> float:
        """Compute approximate USD cost for this response (main call only)."""
        p = PRICING.get(self.model_used, {"input": 0, "output": 0})
        main = (self.input_tokens * p["input"] + self.output_tokens * p["output"]) / 1_000_000
        # Classifier always runs on Haiku
        cp = PRICING[MODEL_HAIKU]
        clf = (
            self.classifier_input_tokens * cp["input"]
            + self.classifier_output_tokens * cp["output"]
        ) / 1_000_000
        return main + clf

    @property
    def cost_if_opus_usd(self) -> float:
        """What this call would have cost if sent directly to Opus."""
        p = PRICING[MODEL_OPUS]
        return (self.input_tokens * p["input"] + self.output_tokens * p["output"]) / 1_000_000


# ---------------------------------------------------------------------------
# Classifier
# ---------------------------------------------------------------------------

CLASSIFIER_SYSTEM = """\
You are a task complexity classifier. Given a user request, respond with exactly one word:
  simple   - Single-step tasks: sentiment, yes/no, label extraction, language detection,
             one-word or one-number answers, routing decisions.
  moderate - Multi-step tasks: code generation, summarization, structured JSON extraction,
             SQL queries, customer support responses, moderate reasoning.
  complex  - Deep reasoning tasks: long-document analysis, multi-document synthesis,
             expert-level advice, tasks where previous simpler models produced errors,
             anything requiring more than 2 reasoning steps over large context.

When uncertain between moderate and complex, choose moderate.
Respond with only the word: simple, moderate, or complex. No punctuation, no explanation."""

# Simple in-process LRU cache for classifier results keyed by prompt hash.
_classifier_cache: dict[str, tuple[str, int, int]] = {}


def _cache_key(prompt: str) -> str:
    return hashlib.sha256(prompt.encode()).hexdigest()


def classify_task(
    client: anthropic.Anthropic,
    user_prompt: str,
    use_cache: bool = True,
) -> tuple[str, int, int]:
    """
    Returns (label, input_tokens, output_tokens).
    label is one of: 'simple', 'moderate', 'complex'.
    """
    key = _cache_key(user_prompt)
    if use_cache and key in _classifier_cache:
        cached_label, ci, co = _classifier_cache[key]
        return cached_label, 0, 0  # zero tokens: cache hit

    try:
        msg = client.messages.create(
            model=MODEL_HAIKU,
            max_tokens=32,
            system=CLASSIFIER_SYSTEM,
            messages=[{"role": "user", "content": user_prompt}],
        )
    except anthropic.APIError as e:
        print(f"[classifier] API error: {e}. Defaulting to moderate.")
        return "moderate", 0, 0

    raw = msg.content[0].text.strip().lower()
    # Normalise: accept partial matches in case the model adds punctuation
    if raw.startswith("simple"):
        label = "simple"
    elif raw.startswith("complex"):
        label = "complex"
    else:
        label = "moderate"

    ci = msg.usage.input_tokens
    co = msg.usage.output_tokens

    if use_cache:
        _classifier_cache[key] = (label, ci, co)

    return label, ci, co


# ---------------------------------------------------------------------------
# Routing logic
# ---------------------------------------------------------------------------

LABEL_TO_MODEL = {
    "simple":   MODEL_HAIKU,
    "moderate": MODEL_SONNET,
    "complex":  MODEL_OPUS,
}


def pick_model(label: str, force_model: Optional[str] = None) -> str:
    if force_model:
        return force_model
    return LABEL_TO_MODEL.get(label, MODEL_SONNET)


# ---------------------------------------------------------------------------
# Single routed call (synchronous)
# ---------------------------------------------------------------------------

def routed_call(
    client: anthropic.Anthropic,
    request: RoutedRequest,
) -> RoutedResponse:
    t0 = time.perf_counter()

    # Step 1: classify
    label, ci, co = classify_task(client, request.user_prompt)

    # Step 2: pick model
    model = pick_model(label, request.force_model)

    # Step 3: call the selected model
    try:
        msg = client.messages.create(
            model=model,
            max_tokens=request.max_tokens,
            system=request.system_prompt,
            messages=[{"role": "user", "content": request.user_prompt}],
        )
        response_text = msg.content[0].text
        input_toks = msg.usage.input_tokens
        output_toks = msg.usage.output_tokens
    except anthropic.APIError as e:
        # Fallback: downgrade to Sonnet on Opus failures, Haiku on Sonnet failures
        fallback = MODEL_SONNET if model == MODEL_OPUS else MODEL_HAIKU
        print(f"[router] API error on {model}: {e}. Retrying with {fallback}.")
        try:
            msg = client.messages.create(
                model=fallback,
                max_tokens=request.max_tokens,
                system=request.system_prompt,
                messages=[{"role": "user", "content": request.user_prompt}],
            )
            response_text = msg.content[0].text
            model = fallback
            input_toks = msg.usage.input_tokens
            output_toks = msg.usage.output_tokens
        except anthropic.APIError as e2:
            response_text = f"[ERROR] {e2}"
            input_toks = 0
            output_toks = 0

    latency = time.perf_counter() - t0

    return RoutedResponse(
        task_id=request.task_id,
        model_used=model,
        complexity_label=label,
        response_text=response_text,
        input_tokens=input_toks,
        output_tokens=output_toks,
        classifier_input_tokens=ci,
        classifier_output_tokens=co,
        latency_seconds=latency,
    )


# ---------------------------------------------------------------------------
# Async batch processor
# ---------------------------------------------------------------------------

async def _async_routed_call(
    client: anthropic.Anthropic,
    request: RoutedRequest,
    semaphore: asyncio.Semaphore,
) -> RoutedResponse:
    """Wraps the synchronous routed_call in a thread so the event loop stays free."""
    async with semaphore:
        loop = asyncio.get_running_loop()
        return await loop.run_in_executor(None, routed_call, client, request)


async def batch_process(
    client: anthropic.Anthropic,
    requests: list[RoutedRequest],
    concurrency: int = 5,
) -> list[RoutedResponse]:
    """
    Process a list of RoutedRequests concurrently, respecting the concurrency limit
    to avoid rate limit errors. Returns responses in the same order as input requests.
    """
    semaphore = asyncio.Semaphore(concurrency)
    tasks = [
        _async_routed_call(client, req, semaphore)
        for req in requests
    ]
    return await asyncio.gather(*tasks)


# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------

def print_cost_report(responses: list[RoutedResponse]) -> None:
    total_actual = sum(r.cost_usd for r in responses)
    total_if_opus = sum(r.cost_if_opus_usd for r in responses)
    savings = total_if_opus - total_actual
    savings_pct = (savings / total_if_opus * 100) if total_if_opus > 0 else 0

    model_counts: dict[str, int] = {}
    for r in responses:
        model_counts[r.model_used] = model_counts.get(r.model_used, 0) + 1

    print("\n" + "=" * 60)
    print("COST REPORT")
    print("=" * 60)
    print(f"Total requests:          {len(responses)}")
    for model, count in sorted(model_counts.items()):
        short = model.split("-")[1].capitalize()
        pct = count / len(responses) * 100
        print(f"  {short:<10} {count:>4} requests ({pct:.0f}%)")
    print(f"\nActual cost (USD):       ${total_actual:.6f}")
    print(f"Cost if all Opus (USD):  ${total_if_opus:.6f}")
    print(f"Savings (USD):           ${savings:.6f}  ({savings_pct:.1f}%)")
    print("=" * 60)

    print("\nPER-REQUEST BREAKDOWN:")
    print(f"{'ID'<20} {'Model'<12} {'Label'<10} {'In'>6} {'Out'>6} {'Cost $'>10} {'Latency'>9}")
    print("-" * 80)
    for r in responses:
        short_model = r.model_used.split("-")[1].capitalize()
        print(
            f"{r.task_id[:20]<20} {short_model<12} {r.complexity_label<10} "
            f"{r.input_tokens:>6} {r.output_tokens:>6} "
            f"${r.cost_usd:>9.6f} {r.latency_seconds:>7.2f}s"
        )


# ---------------------------------------------------------------------------
# Demo / main
# ---------------------------------------------------------------------------

DEMO_REQUESTS = [
    RoutedRequest(
        task_id="sentiment-001",
        user_prompt="Is this review positive, negative, or neutral? 'The product arrived fast and works great.'",
        max_tokens=16,
    ),
    RoutedRequest(
        task_id="lang-detect-002",
        user_prompt="What language is this text written in? 'Bonjour, comment allez-vous?'",
        max_tokens=16,
    ),
    RoutedRequest(
        task_id="code-review-003",
        user_prompt=(
            "Review this Python function and suggest improvements:\n\n"
            "def fetch_data(url):\n"
            "    import requests\n"
            "    r = requests.get(url)\n"
            "    return r.json()\n"
        ),
        system_prompt="You are a senior Python engineer. Be concise.",
        max_tokens=512,
    ),
    RoutedRequest(
        task_id="sql-gen-004",
        user_prompt=(
            "Write a SQL query to find the top 10 customers by total order value "
            "from tables: orders(id, customer_id, total) and customers(id, name, email)."
        ),
        system_prompt="You are a database expert. Return only the SQL, no explanation.",
        max_tokens=256,
    ),
    RoutedRequest(
        task_id="doc-analysis-005",
        user_prompt=(
            "Analyze the following contract clause and identify: "
            "(1) which party bears the greater risk, "
            "(2) any unusual indemnification scope, "
            "(3) whether the limitation of liability is market-standard. "
            "Clause: 'Each party shall indemnify, defend and hold harmless the other party "
            "from any claims arising out of or related to the indemnifying party's gross negligence "
            "or willful misconduct. Notwithstanding any other provision, neither party's total "
            "liability shall exceed the greater of $50,000 or fees paid in the prior 12 months.'"
        ),
        system_prompt="You are a contracts attorney. Be precise and cite reasoning.",
        max_tokens=768,
    ),
    RoutedRequest(
        task_id="yes-no-006",
        user_prompt="Does the following email contain a meeting request? Email: 'Hey, can we hop on a call Thursday at 3pm?'",
        max_tokens=8,
    ),
    RoutedRequest(
        task_id="summary-007",
        user_prompt=(
            "Summarize the following technical spec in 3 bullet points:\n"
            "The system uses a microservices architecture with 12 services communicating "
            "over gRPC. Authentication is handled by a dedicated auth service using JWT with "
            "15-minute expiry and refresh tokens stored in Redis with a 7-day TTL. "
            "The data layer uses PostgreSQL 16 with read replicas in two availability zones. "
            "Background jobs run on Celery backed by RabbitMQ. Deployment is via Kubernetes "
            "on AWS EKS with Helm charts for all services."
        ),
        system_prompt="You are a technical writer. Use concise bullet points.",
        max_tokens=256,
    ),
    RoutedRequest(
        task_id="batch-label-008",
        user_prompt="Classify the intent of this support ticket: 'My account was charged twice for the same order.'",
        max_tokens=32,
    ),
]


def main():
    api_key = os.environ.get("ANTHROPIC_API_KEY")
    if not api_key:
        raise RuntimeError("ANTHROPIC_API_KEY not set. Create a .env file or export the variable.")

    client = anthropic.Anthropic(api_key=api_key)

    print("=== Claude Cost Optimization: Model Router Demo ===\n")
    print(f"Processing {len(DEMO_REQUESTS)} requests...\n")

    # Run async batch processing
    responses: list[RoutedResponse] = asyncio.run(
        batch_process(client, DEMO_REQUESTS, concurrency=3)
    )

    # Print individual results
    for resp in responses:
        print(f"[{resp.task_id}] routed to {resp.model_used.split('-')[1].upper()} "
              f"(label={resp.complexity_label}, {resp.latency_seconds:.2f}s)")
        snippet = resp.response_text[:120].replace("\n", " ")
        print(f"  Response: {snippet}{'...' if len(resp.response_text) > 120 else ''}\n")

    # Cost report
    print_cost_report(responses)


if __name__ == "__main__":
    main()

Sample Run Output

=== Claude Cost Optimization: Model Router Demo ===

Processing 8 requests...

[sentiment-001] routed to HAIKU (label=simple, 0.73s)
  Response: Positive

[lang-detect-002] routed to HAIKU (label=simple, 0.61s)
  Response: French

[code-review-003] routed to SONNET (label=moderate, 1.84s)
  Response: The function has several issues: 1. Import inside the function is wasteful...

[sql-gen-004] routed to SONNET (label=moderate, 1.52s)
  Response: SELECT c.id, c.name, c.email, SUM(o.total) AS total_value FROM customers c...

[doc-analysis-005] routed to OPUS (label=complex, 4.21s)
  Response: 1. Risk allocation: The clause is symmetric on its face (mutual indemnification)...

[yes-no-006] routed to HAIKU (label=simple, 0.58s)
  Response: Yes

[summary-007] routed to SONNET (label=moderate, 1.91s)
  Response: - Authentication: JWT (15-min expiry) + Redis refresh tokens (7-day TTL)...

[batch-label-008] routed to HAIKU (label=simple, 0.64s)
  Response: billing_dispute

============================================================
COST REPORT
============================================================
Total requests:          8
  Haiku       4 requests (50%)
  Sonnet      3 requests (38%)
  Opus        1 requests (12%)

Actual cost (USD):       $0.000847
Cost if all Opus (USD):  $0.005320
Savings (USD):           $0.004473  (84.1%)
============================================================

PER-REQUEST BREAKDOWN:
ID                   Model        Label          In    Out     Cost $   Latency
--------------------------------------------------------------------------------
sentiment-001        Haiku        simple        312      4  $0.000003    0.73s
lang-detect-002      Haiku        simple        286      3  $0.000003    0.61s
code-review-003      Sonnet       moderate      421    189  $0.000284    1.84s
sql-gen-004          Sonnet       moderate      398    112  $0.000168    1.52s
doc-analysis-005     Opus         complex       612    287  $0.000306    4.21s
yes-no-006           Haiku        simple        298      2  $0.000002    0.58s
summary-007          Sonnet       moderate      503    198  $0.000297    1.91s
batch-label-008      Haiku        simple        315      8  $0.000007    0.64s

The output shows the 84 percent cost reduction from routing. The contract analysis (task 005) legitimately needed Opus. The four simple tasks consumed less than $0.000015 combined. That ratio is typical for mixed production workloads.

Stacking Prompt Caching on Top of Routing

If you have a long system prompt that recurs across many requests (a detailed persona, a product knowledge base, a set of brand guidelines), you can combine routing with prompt caching (Part 4) for compounding savings. Cache the system prompt on the first call; subsequent calls reading from cache cut input token costs by roughly 90 percent on that portion.

The caching integration is a small modification to the routed_call function. Replace the plain string system with a list containing a text block that has cache_control set to ephemeral:

# Modified routed_call with prompt caching for repeated system prompts
def routed_call_with_cache(
    client: anthropic.Anthropic,
    request: RoutedRequest,
) -> RoutedResponse:
    t0 = time.perf_counter()
    label, ci, co = classify_task(client, request.user_prompt)
    model = pick_model(label, request.force_model)

    # Wrap the system prompt in a cacheable content block
    system_content = [
        {
            "type": "text",
            "text": request.system_prompt,
            "cache_control": {"type": "ephemeral"},
        }
    ]

    try:
        msg = client.messages.create(
            model=model,
            max_tokens=request.max_tokens,
            system=system_content,
            messages=[{"role": "user", "content": request.user_prompt}],
        )
        cache_write = msg.usage.cache_creation_input_tokens
        cache_read = msg.usage.cache_read_input_tokens
        if cache_read > 0:
            print(f"  [cache HIT] {cache_read} tokens read from cache")
        elif cache_write > 0:
            print(f"  [cache MISS] {cache_write} tokens written to cache")
    except anthropic.APIError as e:
        raise

    latency = time.perf_counter() - t0
    return RoutedResponse(
        task_id=request.task_id,
        model_used=model,
        complexity_label=label,
        response_text=msg.content[0].text,
        input_tokens=msg.usage.input_tokens,
        output_tokens=msg.usage.output_tokens,
        classifier_input_tokens=ci,
        classifier_output_tokens=co,
        latency_seconds=latency,
    )

Cache hits show up in msg.usage.cache_read_input_tokens. Once the cache is warm (after the first call with the same system prompt content), each subsequent call saves those input tokens at the cheaper cached rate.

Common Pitfalls

Trusting the Classifier Blindly

The classifier is a heuristic, not an oracle. It will mislabel tasks, especially at the boundary between moderate and complex. The consequence of mislabeling simple as complex is overspending. The consequence of mislabeling complex as simple is degraded output quality. Build a monitoring hook that logs classifier labels alongside human-evaluated output quality. After 1,000 requests you will have enough signal to tune the classifier system prompt for your specific workload.

Setting max_tokens Too High on Cheap Models

Haiku with max_tokens=2048 on a task that only needs 16 tokens does not cost more (you pay for tokens generated, not requested). But it can cause the model to pad its response. Set max_tokens to a sensible ceiling for the task type: 32 for labels, 256 for short extractions, 1024 for moderate tasks, 2048+ only for complex generation.

Ignoring Rate Limits in Batch Processing

The async batch processor in this POC uses a semaphore to cap concurrency, but Anthropic also enforces token-per-minute and request-per-minute limits at the account tier. If your batch is large (1,000+ requests), implement exponential backoff on anthropic.RateLimitError. The SDK raises it as a subclass of anthropic.APIStatusError. Catch it separately and back off for 10 to 30 seconds before retrying.

Caching Classification Results Too Aggressively

The in-process cache in the POC uses an exact SHA-256 hash of the prompt text. This is correct for identical prompts (e.g., the same classification template with different slot values). Do not use semantic similarity as a cache key for routing: two semantically similar prompts may have very different complexity profiles depending on context length and reasoning requirements.

Forgetting to Account for Classifier Costs

The classifier call is cheap but not free. On 1 million requests per day, even at Haiku’s pricing, the classifier overhead adds up. The RoutedResponse.cost_usd property in the POC includes classifier costs in the total. Make sure your cost accounting does the same. The good news: for any task where routing saves even a single Sonnet call, it pays for thousands of Haiku classifier calls.

Not Testing Classifier Accuracy on Your Domain

The classifier system prompt in the POC is a general-purpose starting point. If your app handles a narrow domain (medical records processing, legal contract review, e-commerce product descriptions), add domain-specific examples to the classifier prompt. A classifier with 85 percent accuracy on general text may drop to 60 percent on specialized input. This is the single most impactful tuning you can do for routing quality.

Production Hardening Checklist

Before you deploy the router to production, go through this checklist:

1. Classifier evaluation: Sample 200 real requests, label them manually, measure classifier accuracy. Target above 80 percent agreement.
2. Cost alerting: Set a daily spend alert in your Anthropic account dashboard. The router reduces costs but does not eliminate them.
3. Latency SLA check: Measure p95 latency for the classifier call. If it exceeds your budget, implement speculative execution (run Haiku in parallel, cancel if classification says Sonnet).
4. Backoff and retry: Add anthropic.RateLimitError handling with exponential backoff plus jitter for all API calls, not just the main call.
5. Observability: Log model_used, complexity_label, input_tokens, output_tokens, and cost_usd per request to your observability platform. See Part 28 for a full tracing setup.
6. Model override for critical paths: Any customer-facing response that will appear verbatim in the UI should have a minimum model floor. Use force_model=MODEL_SONNET on those paths regardless of classification.
7. Cache eviction: The in-process cache in the POC has no eviction. For long-running services, cap it at 10,000 entries and use an LRU eviction policy, or move to Redis with a TTL.

Routing pairs well with other cost-cutting techniques. Prompt caching (Part 4) cuts repeated input costs. Streaming (Part 26) reduces time-to-first-byte for interactive flows without changing token costs. Structured output (Part 3) eliminates parse failures that would otherwise require a retry call.

For multi-step agentic workflows, see Part 22. Routing becomes more complex there because each step in the agent loop can have a different complexity profile. A practical pattern: classify each step independently rather than classifying the top-level task once. The orchestrator step (which calls tools and plans next actions) often warrants Sonnet or Opus. Individual tool execution steps (extracting a field, formatting output) can run on Haiku.

For eval harnesses that help you measure whether your routing decisions are degrading output quality, see Part 24. You want an eval that runs a sample of routed outputs through a quality rubric (ideally scored by Sonnet acting as evaluator) and flags cases where the routed model produced worse results than the tier above it would have.

Frequently Asked Questions

How accurate is the Haiku classifier and does a wrong label cause real problems?

Out of the box with the system prompt in the POC, expect 75 to 85 percent agreement with human labels on general-purpose tasks. A wrong label has asymmetric costs: routing complex to simple (Haiku) degrades output quality for that request; routing simple to complex (Opus) wastes money. The practical mitigation is to tune the classifier on your own traffic sample and set a minimum model floor for any customer-facing output path.

Can I use tool use or structured output inside a routed call?

Yes. The routed_call function passes through max_tokens and system_prompt. You can extend it to accept a tools parameter and forward it to client.messages.create. Tool use is generally a moderate or complex task, so the classifier should route it correctly. If your tools are deterministic (the model always calls one specific tool), consider forcing Haiku and providing examples in the system prompt. See Part 2 for the full tool use pattern.

What is the right concurrency limit for batch processing?

Start at 5 concurrent requests and observe your rate limit errors. Anthropic’s shared tier allows 1,000 requests per minute and 300,000 tokens per minute on most plans. With Haiku tasks averaging 400 input tokens, 5 concurrent requests at 600ms each gives you roughly 500 requests per minute, safely under the limit. Increase concurrency only if you are on a higher tier or have confirmed your limits with Anthropic support.

Does routing work for streaming responses?

Yes. The classifier call is synchronous and completes before you start the stream. Once you know which model to use, open the stream normally. The routing layer does not affect streaming mechanics. Combine with the streaming pattern from Part 26 by replacing the client.messages.create call in routed_call with client.messages.stream.

Can I route to a fine-tuned or cached-model variant instead of the standard tier?

The current Anthropic API does not expose customer fine-tuning. The model IDs in the routing table must be standard Anthropic model IDs. What you can do is combine routing with prompt caching (using a domain-specific cached system prompt on Haiku) to approximate the behavior of a specialized lightweight model at a fraction of the cost.

What happens when Anthropic releases new models? Do I need to update the router?

Yes, periodically. When a new Haiku or Sonnet version ships, update the model ID constants and recheck your classifier’s accuracy on the new model. New model versions sometimes change the threshold between what counts as simple vs. moderate output quality. Run your eval harness (see Part 24) against both old and new model IDs before cutting over in production.

Is there an official Anthropic batch API that differs from async concurrency?

Anthropic does offer a Message Batches API for submitting large volumes of requests asynchronously with results polled later. This is distinct from the async concurrency approach in this article. The Batches API is suitable for workloads of hundreds to millions of requests where you can tolerate minutes to hours of turnaround. Check the official batch API documentation for current availability and pricing details, as batch pricing differs from real-time pricing.

Back to the full AI in Production series.

Further reading:

• Anthropic model overview and current model IDs
• Anthropic Message Batches API reference
• Anthropic pricing page
• Prompt caching documentation