LLM token-cost & context economy — promote the principle, gate the tools

OpenAuthor: @relbnsOwner: openEffort: M

Problem

A round of widely-shared posts (a Netflix engineer cut a ~$280/month Claude bill to ~$110 with an open-source compression tool) put a real number on a real cost: most of an AI coding agent's bill is not the developer's questions or code, it is the "junk tokens" that tools append to every request — verbose tool output, whole log blocks, hundreds of DB rows when three were needed, code the model already saw.

The platform already has a context-economy stance at the artifact layer (lazy loading: ~50KB/session; singalong measured ~15k→~9k tokens/task, −40%). It has no stance at the agent-runtime I/O layer, where this junk accumulates. The efficiency/cost discipline was opened as INS-1 (docs/insights/index.md, priority quality > speed > cost, "code before model") but was never promoted to a methodology section, and the two gates it calls for do not exist yet. So today the decision "how do we cut token cost without losing output quality?" has no documented, deterministic answer — exactly the gap a methodology is supposed to close.

The two tools the posts recommend — Headroom (chopratejas/headroom, ~16.2k★) and RTK (rtk-ai/rtk, ~59.5k★) — are tempting shortcuts. But both are Apache-2.0, ~5-month-old self-classified beta, with vendor-self-reported savings and no independent benchmarks, and they are lossy. RTK additionally has an open security review (issue #640: shell-injection, telemetry default-on in the shipped binary, exit codes not propagated so an agent can misread a failed command as success, command lines incl. secrets retained in a local tracking.db for 90 days). Adopting one by default would conflict with the platform's quality-first ordering.

Proposal

Accept the diagnosis, reject off-the-shelf adoption. Promote the principle, don't bolt on a tool.

A lossy rewriting proxy in front of Claude gambles correctness exactly where it matters most (multi-step reasoning, math, long-context aggregation) and can defeat prompt caching — changing the prompt prefix turns 10%-price cache reads back into full-price reprocessing. A pre-registered RCT on Claude Sonnet 4.5 (358 runs) found moderate compression cut cost ~28% but aggressive compression raised cost ~1.8% (the model expanded its output to compensate). Meanwhile Anthropic's own first-party stack independently validates a lossless-first ordering: prompt caching (lossless, "response is identical"), context-editing (deterministic, zero-inference clearing of finished tool results), the memory tool (offload state verbatim), Batch API (flat −50%), model tiering — and it positions model-based summarization (compaction) last, only for content that cannot be re-fetched.

So the deliverable is to promote INS-1 into a methodology section that codifies a Cost-Lever Tier Ladder, wire the two gates INS-1 already names, and record the Headroom/RTK evaluation as a decision so it is not re-litigated. The tools become a gated, validated option on output we do not control — never a default.

The Cost-Lever Tier Ladder

Tier	Lever	Lossy?	Stance
0	Source-side deterministic pruning — return only the needed fields/rows/lines	No	Lead here. Pure "code before model"; fixes the cause.
1	First-party lossless levers — prompt caching, context-editing (clear finished tool results), memory tool, Batch API, model tiering	No	Adopt. The safe cost wins. claude-api skill already mandates caching.
2	Deterministic selective stripping of uncontrolled output (RTK-style) — preserve errors/diffs/exit codes	Yes (bounded)	Gated pilot only; off correctness-critical paths; local build, telemetry off, validated against /cost.
3	Lossy model-based compression (Headroom ML path, LLMLingua) / semantic caching (GPTCache)	Yes	Last resort; never on reasoning-critical live context; requires preserved source-of-truth + task validation.

Two cross-cutting rules: (a) choose the lowest tier that solves the problem; never trade up a tier for cost if it touches reasoning-critical context. (b) the cache-defeat warning — a prompt-rewriting proxy can bust prompt caching and raise net cost.

Why now / Why us

• The principle is already half-written. INS-1 exists; this just promotes it and gives the team a decision framework instead of an ad-hoc choice. Low marginal cost.
• It is on-philosophy. This is "code before model" + quality-first applied to token cost — and the model vendor (Anthropic) endorses the exact ordering, so we are not betting on a fad.
• It pre-empts a bad shortcut. Without a documented stance, the next person under cost pressure may just drop a lossy proxy in front of Claude — which can hurt both quality and cost. The proposal turns that into a gated, validated path.
• Not urgent. Captured for "when we get to it" — there is no incident forcing it; it is an optimization-discipline investment.

Sketch of approach

• New methodology doc docs/methodology/context-and-cost-economy.md — promotes INS-1; codifies the Tier Ladder + the two cross-cutting rules; includes an "Evaluated tools" subsection recording the Headroom/RTK findings & verdict. Cross-link from deterministic-ai.md (principle 2) and the efficiency/cost discipline.
• Wire INS-1's two missing gates:
  • "Can this be deterministic?" gate in the new-artifact skill (.claude/skills/new-artifact/SKILL.md, ~step 3) — serves Tier 0.
  • Efficiency lens (new check) in the code-reviewer agent (.claude/agents/code-reviewer.md) — flags: returns only needed fields, no unbounded context, lossless levers before lossy, appropriate model tier.
• Reinforce two existing artifacts: extend the ai-llm agent token-aware checklist with the context-editing / memory / batch levers + the "no rewriting proxy in front of the cache" rule; add the cache-defeat caveat to the claude-api skill next to its prompt-caching mandate.
• Record + index: mark INS-1 promoted in docs/insights/index.md; optionally capture the tool decision as an ADR (/decision); run pnpm capabilities so the new doc registers.
• Explicitly NOT doing: no default adoption of Headroom/RTK; no proxy in front of Claude by default; no lossy compression on reasoning-critical context; no semantic caching for correctness-critical flows.
• If a Tier-2 pilot is ever greenlit: measure against Claude Code /cost on a real session (not vendor numbers) AND run a quality check on a correctness-sensitive task; adopt only if cost drops and quality holds. Local build (no compiled telemetry endpoint), off tests/lint exit-status, never where command lines carry secrets.

Open questions

• Does the tier ladder live as its own context-and-cost-economy.md, or fold into an expanded deterministic-ai.md principle-2 section? (Leaning standalone, cross-linked.)
• Is the tool evaluation worth a full ADR, or is the "Evaluated tools" subsection in the methodology doc enough?
• Should a deterministic "source-side pruning" lint/helper be offered for our own MCP tool handlers (the Tier-0 enabler), or is that a follow-up proposal once the principle lands?

Related

• Source posts: @AlexTseitlin, @Parzival_1991, @yonin44.
• Tools evaluated: chopratejas/headroom, rtk-ai/rtk (security review #640).
• Anthropic first-party levers: prompt caching, context editing, memory tool, effective context engineering.
• INS-1 in docs/insights/index.md; the claude-api skill and ai-llm agent (existing token/cost owners).
• Full session research + adversarial verification: workflow run wf_4df52353-4c4 (Headroom, RTK, Anthropic stance, alternatives incl. LLMLingua/RouteLLM/GPTCache, plus tool-reality and quality-tradeoff verdicts).

---

Triage notes

To be filled in at the next triage session.