LLM Cost Audit

---
name: llm-cost-audit
description: Audit and fix the code patterns that make AI/LLM API bills blow up: retries that loop or stack, no spending limit, no cap on output length, agent/tool loops that never stop, rate-limit pileups (429s), and re-sending the same big prompt without caching. Provider- and language-agnostic — OpenAI, Anthropic, Gemini/Vertex, Bedrock, Azure, Mistral, Cohere, and gateways/frameworks like OpenRouter, LiteLLM, LangChain, and the Vercel AI SDK. Use whenever the user raises AI/LLM cost or a surprise bill: "my OpenAI/Anthropic/Gemini bill spiked", "AI costs out of control", "runaway spend", "retry storm", "getting 429s", "add a spend cap or token budget", "reduce AI API costs", "audit our AI integration", "is this going to bankrupt us" — or when they just describe the symptom (a cost blowout, a loop that wouldn't stop, a scary invoice) without naming the cause. Not for general cloud/infra cost or for improving answer quality. Works in any agent host.
---

# LLM Cost Audit

You are auditing a codebase's AI/LLM integrations for the specific bugs that turn a normal month into a five-figure bill. LLM spend is dangerous in a way most cloud cost isn't: a single defect can run *unbounded* overnight. A retry loop on a `429` hammers the provider thousands of times; an agent with no iteration cap calls tools until it dies of old age; a model with no output cap falls into a repetition loop and bills to the context limit; one user who scripts your endpoint runs up the bill for everyone. None of these throw an error in code review — they just cost money.

The bar is **"would this surprise me on the invoice?"**, not "is this theoretically optimal." A weekend project calling OpenAI once per request doesn't need a distributed rate limiter. But *any* product that bills real money to an AI provider needs a ceiling on spend, a cap on output, and retries that can't turn into a storm. Match the bar of the surrounding code: don't demand a token-budget service in a 200-line side project, but do flag the side project that retries forever on a 500.

**Don't break working features.** Never propose a fix that breaks a working flow without flagging it. Prefer additions (add a limit, add a spending check) over changes that could make output worse or lock users out — like cutting an output limit below what a feature needs, swapping the model, or setting a rate limit so low that real users get blocked. Every finding flags what to watch when applying the fix.

The audit produces **one short report**, worst-first — the thing that can run up an unlimited bill comes before the thing that wastes 10%.

## Write it for a working developer

The person reading this is a good developer who has probably never tuned an AI integration for cost, doesn't know the jargon, and doesn't need to. Your job is to make them go "oh, I see it — and I know what to change." Accuracy and brevity both matter; a sharp three-sentence finding beats a correct but exhausting one.

- **Plain words, not insider terms.** Don't write "thundering herd," "backpressure," "idempotent," "poison job," "exponential backoff with jitter," "quadratic," "RPM/TPM." Say the actual thing: "all the retries fire at the same instant and make the overload worse," "the same input always gives the same answer, so you can save it and skip the call," "a background job that fails and keeps re-running itself forever," "wait a little longer after each failed try, with some randomness so they don't all retry at once," "the provider's limit on requests per minute." If a precise term truly helps, explain it once in plain words and move on.
- **Short.** A couple of tight sentences per finding. Cut the warm-up, the hedging, and the repetition.
- **No made-up dollar figures.** You don't have their bill, so invented "$7,500–$50,000" ranges with disclaimers are just noise. Say plainly what has no limit and why ("nothing stops this from running all night"). Use a real number only when it's a simple, honest comparison ("a smaller model does this for a fraction of the price").
- **Every finding ends with something to do.** The fix is the point — concrete enough to act on without more research.

**Plain English is for the explanation, not the code.** The developer reads the *why*; the *fix* gets implemented — often by an AI coding agent working straight from this report. So keep the explanation jargon-free, but make the fix itself technically exact: the real parameter name for the actual provider (`max_tokens` vs `maxOutputTokens` vs `maxTokens`), real retry/caching/job code, exact config keys, correct status codes. Never simplify a snippet to sound friendlier — a wrong-but-approachable fix is worse than useless. Plain words for the "why," full precision for the "what to change." When the precise detail differs by provider, pull it from `references/providers.md` rather than guessing.

## Mode and scope

- If `$ARGUMENTS` is a path, audit only that path. Otherwise audit the whole repo, excluding: `node_modules`, `.git`, `dist`, `build`, `out`, `.next`, `vendor`, `target`, `__pycache__`, `.venv`, lock files, minified assets, and anything in `.gitignore` that looks like build output.
- **Announce the scope before scanning** so the user can redirect you.
- **Detect the stack and the providers early.** Look at manifests (`package.json`, `requirements.txt`/`pyproject.toml`, `Gemfile`, `go.mod`, `Cargo.toml`, `composer.json`) for AI SDKs, then run the discovery scan. The provider matters less than you'd think — the failure modes below are nearly identical across all of them — but it tells you which parameter names and caching APIs to check (see `references/providers.md`).

## Step 1 — Find every LLM call site

You cannot audit calls you haven't found. Run the bundled scan, which greps for the signatures of every major provider, gateway, and framework so you don't have to reconstruct the patterns each time:

```bash
python3 scripts/find_llm_callsites.py            # scans cwd
python3 scripts/find_llm_callsites.py path/to/src
```

It prints `file:line` for each match grouped by provider/framework, plus a summary of which integrations exist. Treat its output as your work-list. If it finds nothing but you have reason to believe there are AI calls (e.g. an `OPENAI_API_KEY` in `.env`), the integration may be behind a gateway or a thin wrapper — grep for the base URLs and env var names listed in `references/providers.md` and for the project's own client wrapper (`ai_client`, `llm.`, `chat(`, `complete(`).

Read the actual call sites the scan surfaces. The bugs live in *how* each call is configured and *what surrounds it* (the retry wrapper, the job that enqueues it, the loop it sits in), not in the SDK import.

## Step 2 — Audit each finding two ways

Two passes, because the most expensive guardrails are **absences across the whole system**, not bugs at a single line.

1. **Per-call-site** — walk each call the scan found and check it against the checklist below.
2. **System-level** — ask whether each *capability* exists *anywhere*. Is there any global spend ceiling? Any per-user/per-tenant cap? Any client-side concurrency limit? Any usage logging? A missing system-wide ceiling is usually the single most important finding, and no individual call site reveals it — you find it by searching for evidence it exists and reporting if it doesn't (see the System-level section).

Before flagging, confirm. If a call looks uncapped, check for a wrapper or middleware that caps it. If retries look infinite, check the SDK's defaults (most SDKs retry 2 times by default — your custom wrapper on *top* of that is the multiplicative bug). Speculation isn't a finding; a traced call path is.

---

## The checklist

Each item: how to detect it, **why it costs money** (the concrete scenario), and the bad → good shape of the fix. Severity comes from this section and drives report order.

### Critical — can cause unbounded or runaway spend

| Issue | Detection hint | Why it costs / Bad → Good |
|---|---|---|
| **No ceiling on total spend anywhere** | Search the whole repo for any budget/quota/spend gate: `budget`, `quota`, `spend`, `cost_limit`, `usage_limit`, `credits`, a daily/monthly counter checked before calls. If nothing gates calls on cumulative cost, this is the finding. | One scripted user, one runaway loop, or one viral day bills with no upper bound. → A budget check before the call (per-user *and* global) that refuses or queues once a ceiling is hit, plus an alert. |
| **Retrying errors that can never succeed** | Find the retry code. Does it retry on *every* error, or only the ones worth retrying (`429` rate-limit, `500`/`503` server errors, timeouts)? Retrying a `400 bad request` or `401 unauthorized` will fail every time. | Every retry of a broken request still costs money and still fails. A bad prompt retried 5 times = 5× the cost for zero result. → Retry only rate-limit, server, and timeout errors; for the rest, fail right away. |
| **Retries that fire too fast and all at once** | Look for `for`/`while` retry loops, `sleep(constant)`, or retry libs set to a fixed delay. A growing wait that's missing the bit of randomness is a smaller version of the same problem. | A `429` error means "you're sending too fast." Retrying immediately, or on a fixed timer from many workers at once, means all the retries hit at the same instant — which makes the overload worse and drags it out. → Wait longer after each failed try (e.g. 1s, 2s, 4s) plus a little randomness so they don't all retry in sync, and stop after a few tries. |
| **Retries piled on top of retries** | Your own retry code *plus* the SDK's built-in retries (`max_retries`) *plus* the job queue's retries (Sidekiq/Celery/BullMQ). These multiply together. | 3 layers of 3 retries each = up to 27 paid attempts for one call. → Keep *one* layer of retries and turn the others off. Usually let the SDK handle the quick retries and the job queue handle the give-up case, and delete your own retry loop. |
| **A background job that fails and re-runs itself forever** | Find background jobs that call the AI. When one fails, does it retry with no limit on attempts? Does it re-run even when the thing it's working on was deleted? | A job that always fails (bad input, a deleted record, a too-big document) re-runs every time the queue comes around, forever — and every attempt is a paid AI call. → Limit the number of attempts, give up (and set it aside) after that, and skip the job entirely when the record it needs is gone. |
| **No cap on output tokens** | Check each call for `max_tokens` / `max_output_tokens` / `maxOutputTokens` / `max_completion_tokens` / `maxTokens` (names vary — see `references/providers.md`). Flag calls that omit it. | Output tokens are the expensive ones (often 3–5× input). With no cap, a model that falls into a repetition/degenerate loop generates until it hits the context window, and you pay for every token. → Always set an explicit output cap sized to the feature. |
| **An AI loop that can run forever** | Find `while`/`for` loops that call the model, add the tool result, and call again. Is there a cap on the number of times around, a time limit, and a deadline? | A model that keeps deciding to call one more tool (or two agents handing work back and forth) loops until something else breaks — and each turn re-sends the whole growing conversation, so the cost climbs faster and faster. → Cap the number of loops, set a time limit, and set a total spending limit for the whole run; stop and return what you have when a limit is hit. |
| **Model returns empty or junk output and the code just retries it** | For generation/structured-output calls, is there any check for the model returning nothing, getting cut off at the length limit, or repeating itself before the code retries? | A common blowout: the model returns a bad response, the code counts it as a failure and pays to run it again, over and over, often without recording it. → Detect the bad response (empty, cut off, or repeating), limit how many times you re-run it, and record every attempt so the cost is visible. |