Amazon Drops Claude: Specialized Decision Model Cuts Ticket Costs by 98%

Amazon Drops Claude: Specialized Decision Model Cuts Ticket Costs by 98%

AmazonClaudeStrands AgentsLLMInference Optimization

Sources:Strands Agents 官方博客 + Hacker News

Replacing a flagship foundation model with a lightweight alternative in production typically demands severe compromises in quality. Yet inside Amazon’s core internal ticketing infrastructure, engineering teams pulled off a counterintuitive migration: Claude Sonnet 4.6, long tasked with routing and classification, was stripped from the critical path. Taking its place is Strands Decider, a specialized decision model packing just 4 billion parameters. Per-ticket processing latency plummeted from 4,215 ms to 897 ms, while the inference bill per million tickets collapsed from $22,200 to roughly $500—a 98% reduction.

This precipitous drop in cost stemmed from fundamentally rethinking what classification actually requires. Conventional agent architectures habitually offload classification tasks to generic LLMs, force-feeding them thousands of tokens of policy guidelines and waiting for autoregressive token-by-token generation. But in a closed, single-choice selection task, an orchestration system only requires class probabilities, not conversational text. By abandoning autoregressive decoding altogether, the system eliminated the latency and payload tax of transmitting formatting prompts, while wiping out the need for downstream grammar and JSON parsers.

Tagging Internal Tickets: Burning 6,000 Tokens per Invocation

Amazon operates an immense internal ticketing infrastructure. Every day, floods of inbound tickets covering upgrades, capacity scale-ups, monitoring alerts, and compliance audits must be categorized into one of 20 predefined buckets. Previously, this workload ran on Claude Sonnet 4.6 via Amazon Bedrock, serving as the benchmark choice balancing analytical capability and operational cost.

However, the per-token economics of general-purpose LLMs imposed a punishing tax on high-throughput pipelines. To keep the model strictly aligned with categorization taxonomy, every invocation required a comprehensive system prompt delineating the boundary conditions of all 20 categories, coupled with strict JSON schema constraints. Production telemetry revealed that each ticket consumed an average of 6,554 input tokens and 170 output tokens—totaling 6,724 tokens per run. The overwhelming majority of compute was squandered repeatedly transmitting static taxonomic rules and formatting schemas.

Autoregressively generating tokens over thousands of input tokens crippled turnaround times. Claude’s average latency per ticket hovered around 4,215 ms. Latency exceeding four seconds caused concurrency backlogs on upstream SQS consumer workers. Calculated against published Bedrock rates, pure model inference ran $0.0222 per ticket—translating to a staggering $22,200 bill for every million tickets processed. These mounting operational costs forced the team to rethink the underlying architecture.

Stripping the Generation Pipeline: Outputting Probabilities Slashes Latency by 78%

Faced with a strictly closed classification problem, the team turned to Strands Decider, an open-source decision model. Unlike autoregressive language models, Strands Decider excises the causal language modeling head in favor of a Pointer Head designed specifically to score candidate options. The input consists solely of the current state, the query context, and a discrete array of choices. In a single forward pass, the architecture computes a Softmax across candidate logits, directly emitting class probabilities and calibrated confidence scores.

Strands Decider 架构 Figure: Strands Decider replaces the autoregressive generation head with a Pointer Head to compute candidate option scores. Source: Strands Agents

This architectural shift entirely bypasses the autoregressive sampling loop. Inference executes in one forward pass without iterative decoding delays, and the system no longer needs to defend against malformed JSON or parsing failures. Subsequent benchmarks demonstrated that the E4B variant, fine-tuned on Gemma 4, surpassed the classification accuracy of production Sonnet 4.6. The team approved E4B for direct production rollout.

Pruning inputs and outputs unlocked immediate performance gains. On identical production evaluation datasets, average invocation latency dropped from 4,215 ms to 897 ms—a 78% reduction. Input volume shrank from 6,554 to 2,009 tokens per ticket, while the output collapsed from 170 tokens down to a single option token, cutting total token consumption by 70%. The moment Sonnet 4.6 was shut off, average execution duration for front-end AWS Lambda consumers plunged by 74%.

Strands Decider 延迟分布 Figure: Strands Decider latency distribution across the evaluation set, demonstrating consistently low network round-trips and compute overhead. Source: Strands Agents

The latency drop fundamentally improved serverless resource efficiency. Because Lambda billing scales directly with runtime duration, compressing execution time by three-quarters proportionally decreased the concurrency quota needed to sustain identical message throughput. Queue backlog drained into sub-second intervals, accelerating time-to-mitigation for high-priority operational alarms.

Adopting a lightweight decision model was not a simple drop-in replacement; its restricted context window presented an immediate engineering constraint. Strands Decider enforces a hard ceiling of 4,096 tokens. In production, the original exhaustive definitions and few-shot examples for all 20 categories exceeded 4,000 tokens on their own, leaving zero headroom for the actual ticket contents.

Engineers restructured their prompt engineering methodology from scratch. The team distilled each operational category into one or two sharp decision heuristics, structured as multiple-choice options, and appended a catch-all none fallback option. For unusually lengthy tickets, the pipeline aggressively truncated older historical comments, safeguarding the latest operational status and root-cause fault snippets intact.

Brier 分数变化 Figure: Evolution of decision calibration and accuracy across model iterations; lower Brier scores reflect better probability calibration. Source: Strands Agents

To preserve labeling integrity, the system integrated confidence threshold gates. The team calibrated output probabilities continuously to minimize Brier scores along decision trajectories. Only when the top candidate’s confidence exceeded the threshold did the pipeline apply automated tags. If confidence fell short or resolved to none, the ticket routed gracefully to human engineers for triage.

During end-to-end integration, engineers uncovered a counterintuitive low-level phenomenon: the serialization order of JSON keys altered decision probabilities. Holding key-value contents completely constant while shuffling key order shifted output probabilities by as much as 0.19—enough to flip borderline categorization decisions. The team mitigated this instability by enforcing strict alphabetical key sorting across all inputs, eliminating variance introduced by payload serialization.

AppConfig Three-Way Cutover: Sustaining 5,600 Tickets per Hour on a Single Instance

The updated architecture relies exclusively on standard AWS building blocks. Incoming tickets hit an Amazon SQS queue, triggering Lambda functions to extract payload metadata and invoke an Amazon SageMaker model endpoint. The endpoint provisions an ml.g5.xlarge instance equipped with a single 24GB NVIDIA A10G GPU. The model footprint consumes roughly 19 GiB of VRAM, running in an isolated VPC subnet with zero public outbound internet access.

A single A10G GPU delivered substantial throughput. The endpoint processed individual requests in approximately 640 ms, enabling one compute instance to handle roughly 5,600 tickets per hour. The cluster dynamically auto-scales based on SQS queue depth, streaming logs and metrics into CloudWatch. The entire infrastructure is managed via AWS CDK, partitioned into Beta, Gamma, and Production release stages.

To avoid risky hard cutovers, the architecture integrated dynamic configuration toggles via AWS AppConfig, queried by Lambda on every message pass. The toggle exposed three operational states: BEDROCK routed 100% of traffic to Sonnet 4.6; DUAL shadowed traffic concurrently across both models to diff outputs in real time; and DECIDER transferred complete cutover traffic to the specialized model.

The feature flag provided an ironclad deployment safeguard. Throughout the dual-run phase, routing decisions were still executed against Sonnet outputs while back-end pipelines logged and analyzed discrepancies. This live production mirroring verified generalization under realistic edge cases. In the event of regression, updating AppConfig flipped traffic back to the legacy pipeline within 45 seconds.

Saving $20K per Million Tickets: The Boundaries of Closed-World Selection

On the balance sheet, the unit economics demonstrated a stark divide. Under Bedrock pricing, processing a single ticket cost $0.0222, accumulating to $22,200 per million tickets. Shifting to dedicated SageMaker instances changed the cost structure to fixed compute hours. At us-east-1 on-demand pricing ($1.408/hour), running instances at full saturation cost just $250 per million tickets. Even budgeting a 2x headroom margin for bursty traffic spikes, total expenses leveled out around $500 per million tickets—reducing the per-ticket cost to approximately $0.0005.

Slashing inference costs from $22,000 to $500 represents a 98% savings—roughly a 44x cost efficiency multiplier. These figures reflect self-reported benchmarks from the team’s engineering disclosures using public on-demand rates, without factoring in additional cost reductions from cutting Lambda runtimes by 74%. Across platforms processing tens or hundreds of millions of tickets annually, this architecture frees up substantial operational capital.

However, the success of this architectural shift hinges on strict boundary conditions. The engineers outlined clear prerequisites for applicability: the task must operate as a closed-world selection from known options, volume must be high enough to amortize fixed instance uptime, and downstream workflows must genuinely suffer from legacy generation latency. When all three hold true, specialized decision models deliver extraordinary dividends.

When those conditions break down, the architecture no longer applies. If downstream workflows require explanatory reasoning, if taxonomy taxonomies change weekly, or if essential context cannot fit within a few thousand tokens, decision models cannot substitute for frontier general-purpose LLMs. The high expense of foundation models stems fundamentally from the surplus degrees of freedom retained for open-ended generation. When an engineering task converges into closed-world selection, pruning the autoregressive pipeline and letting a specialized model directly emit candidate probabilities remains the most pragmatic path to scaling AI agents.

Reference Links:

  • Strands Agents Blog: Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
  • Hacker News Discussion: Amazon engineers swap Claude for decision model