Rate-limited by tokens, not requests

An agent run stalled for 27 seconds in the middle of a task this April, and the 429 Too Many Requests behind it pointed everywhere except the real problem. My first theory was the obvious one, too many API calls, but the response headers showed anthropic-ratelimit-requests-remaining: 49 out of 50. The quota that had emptied was input tokens per minute.

Shannon’s agent loop was re-sending the entire context on every turn: an 87-line system prompt, the JSON schemas for all 13 tools, and a listing of every file in the workspace. In a multi-turn agent loop, the budget you’re spending is context replay, not request count, and prompt caching is the baseline that makes the loop affordable, not an optimization to add later.

The runs didn’t fail, which is why the bug hid

The task that surfaced it was as small as tasks get: “tell me another joke”, on a test workspace with about 90 files. Turn 1 returned 200 OK, but the headers on that response already carried the warning: anthropic-ratelimit-input-tokens-remaining: 0. A single request had consumed the entire minute’s input-token quota. The model called its tools (search_content, glob, create_file), the loop went back for turn 2, and the API answered with a 429 and retry-after: 27.

Then the failure vanished. The Anthropic SDK honors retry-after on its own: it waited 27 seconds, retried, and the run completed normally.

The only trace was Retrying request to /v1/messages in 27.000000 seconds, logged at INFO from inside the SDK’s _base_client rather than from my code, while the SSE heartbeat kept the connection alive with a ping every ~15 seconds and nothing in between. From the frontend, multi-turn runs just felt slow. I rate it medium severity for exactly that reason: no hard failure, just a silent ~27-second stall on every rate-limit hit, latent since the project’s inception and first observed on 2026-04-23.

Every turn re-sent the whole context at full price

Each call to stream_turn rebuilt the full request prefix from scratch: the 87-line system prompt, the available-tools section, the schemas for all 13 tools, a canary token (a per-run secret planted in the prompt so leaked instructions can be spotted in the model’s output; it becomes important later), the current-task block, and the entire message history. On a 90-file workspace, the first user message also carries a metadata block listing every file path and UUID, which runs 3-5k tokens before the conversation has said a word.

The arithmetic is short. A single turn costs roughly 15-25k input tokens. The API key’s tier allows 30k input tokens per minute. Two turns inside one minute is a guaranteed 429, on every multi-turn run, regardless of what the user asked for.

None of those bytes needed re-processing. Prompt caching has been in the Anthropic SDK since August 2024, and the installed version (0.79.0) supports it fully; a grep for cache_control across the app returned zero occurrences. Nothing had regressed. The feature was never wired in at all, an absent best practice rather than a broken one, which is the kind of gap no regression test ever flags.

A 429 doesn’t tell you which of three quotas you hit

Anthropic enforces three independent quotas: requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM). Every response carries a separate set of anthropic-ratelimit-* headers for each bucket, ten-plus headers in all. The 429 status only says that you were limited; the header reading *-remaining: 0 says which bucket did it. My debugging rule from this incident: grep the response headers for -remaining: 0 before forming a theory.

Intuition fails here because request counts are the quota everyone knows. Forty-nine of fifty requests remaining looks healthy, and it was. But ITPM is a separate bucket, and on low tiers it’s almost always the tighter constraint for agent workloads: agents replay their whole context every turn. A tier bump raises the ceiling (30k input TPM at Tier 1, 80k at Tier 2, 160k at Tier 3, granted automatically on spend), but it only moves the wall. The replay still grows with every turn of every run.

Cache breakpoints go where the content’s lifetime changes

The fix is Anthropic’s prompt caching, and the design decision is where the markers go. A cache_control marker on a content block tells the API to cache everything up to that point; a later request whose prefix is byte-identical reads it back at roughly a tenth of the input price. The marker position is called a breakpoint, and placing one is a claim about stability: everything before it must not change between requests, or the cache never hits.

Shannon’s context has three layers with three different lifetimes, so the fix ships as a tiered prompt_cache_level setting (an int, 0-3, default 1) rather than a flag.

• Level 1 puts a breakpoint after the system prompt and tool schemas, which are stable across every run and every user.
• Level 2 adds one after the first user message, the workspace context, which is stable per user. Two users hitting the same model share the level-1 cache entry and fork into separate entries here.
• Level 3 keeps a rolling breakpoint on the penultimate message, so each new turn re-processes only the latest message instead of the whole history.

An int instead of a boolean was deliberate. Caching here has four meaningful states: off, system-only, system plus workspace, system plus workspace plus history. A boolean would pick one and force a code change to reach the others; the levels stack, 0 is the escape hatch, and rollout can move up one level at a time.

The API’s own limits shaped the design too: at most 4 breakpoints per request, a 1024-token minimum per cached block, and a 5-minute TTL. Those behavioral limits came from the docs, but the wire shape came from the installed SDK’s type stubs (cache_control_ephemeral_param.py, one ten-line file), which are checked in and can’t drift the way a doc can.

Per-run entropy poisons a cache prefix

There was a trap waiting even after turning caching on. Shannon plants a canary in every run’s system prompt, a fresh secrets.token_hex(16), so output can be scanned for leaked instructions. In executor.py it was concatenated straight onto the prompt:

canary_system_prompt = SYSTEM_PROMPT + tools_section + canary_instruction

That one line makes the system prefix unique per run. Caching switched on naively over it would never hit across runs, and within a run it would only start helping at turn 2. Sixteen random bytes, placed there for a good security reason, zeroing the cache.

The fix is structural: keep the canary in the request, but move it past the breakpoint.

cacheable_system_prefix = SYSTEM_PROMPT + tools_section        # stable across runs
canary_instruction      = build_canary_instruction(canary)     # per-run, uncached

system = [
    {"type": "text", "text": cacheable_system_prefix,
     "cache_control": {"type": "ephemeral"}},                  # cached
    {"type": "text", "text": canary_instruction + task},       # not cached
]

The security posture doesn’t move: the canary is still in every request, scan_text_for_canary still runs against model output, and the outbound prompt order is preserved. Only the block boundary changed, and with it the prefix became byte-identical across runs and users. The rule I wrote down afterward: before anything goes into a system prompt, ask whether it changes per request. Timestamps, request UUIDs, random tokens — if the answer is yes, it belongs after the cache breakpoint, because any one of them converts the hit rate to zero.

What this incident rewrote for me is the unit of budgeting. I used to think in requests, because requests are what shows up in logs, but an agent loop spends tokens, and it spends them on the same bytes over and over: every turn pays again for every turn before it. With the breakpoints in place, turns 2 and later should pay about 10% of the input cost for everything behind the last marker, and the smoke test is already written: confirm cache_creation_input_tokens > 0 on the first request and cache_read_input_tokens > 0 on the second. Until those numbers are in, the claim stands on arithmetic, not measurement.

But the invariant holds either way: if a request is part of a multi-turn loop, it ships with cache_control. Caching is not an optimization. It’s the baseline.