Chatbot LLM Cost: Stateless vs With History

Two chatbots can look identical to users and cost wildly different amounts to run. The difference is one architectural choice: does each message stand alone, or do you re-send the conversation history with every turn? “Remembering” feels free — it’s just more tokens in the prompt — but those tokens compound in a way that surprises almost everyone the first time they see the invoice.

Stateless vs. with history: the headline numbers

A stateless chatbot sends ~800 input / 400 output tokens per message. Add conversation history and the average message carries ~3,000 input / 600 output — same single call, much heavier payload. At 1,000 requests/day (30,000/month), naive monthly cost:

Model                stateless    with history
Claude Sonnet 4.6    $252/mo      $540/mo   (2.1×)
GPT-5.4              $240/mo      $495/mo   (2.1×)
Claude Haiku 4.5     $84/mo       $180/mo   (2.1×)
GPT-5.4 mini         $72/mo       $148/mo   (2.1×)
Gemini 2.5 Flash     $37/mo       $72/mo    (1.9×)
DeepSeek V4 Flash    $7/mo        $18/mo    (2.6×)

So memory roughly doubles the bill at typical conversation lengths. But the flat “2.1×” undersells what’s actually happening — because history doesn’t add a fixed cost per message. It grows.

The real mechanism: conversations cost quadratically

Every turn re-sends everything said so far. If each completed turn adds ~500 tokens of history (a user message plus an assistant reply), the input for turn N is roughly the system prompt plus (N−1) × 500 tokens of transcript. Summed over a conversation, that’s an arithmetic series — total input grows with the square of the turn count:

10-turn conversation (500-token system prompt):
  stateless   10 × 600     =  6,000 input tokens
  with history  Σ per-turn  = 28,500 input tokens   (4.8×)

Turn 2 is cheap. Turn 10 carries the whole transcript.
Turn 30 in a support thread carries a small novel.

This is why “chatbot with history” is a structurally different cost profile, not a variant of the same one: stateless cost scales with message volume, history cost scales with message volume × conversation depth. Long conversations are the expensive ones, and averages hide them.

The good news: history is the perfect caching target

Here’s the part that makes memory affordable. Prompt caching keys on the prompt prefix — and a conversation transcript is append-only. Each turn’s prefix (system prompt + all prior turns) is byte-for-byte what you sent last turn. That’s a near-ideal cache hit pattern, far better than most workloads can hope for.

With cached reads at ~10% of input price (OpenAI, Anthropic, Gemini — 98% off on DeepSeek), the math changes character. If ~80% of your input tokens are cached transcript, effective input cost drops to ~0.28× — and the 4.8× quadratic blowup lands at roughly 1.3× a stateless bot. Memory goes from “doubles the bill” to “adds a third,” purely by structuring the prompt so caching can fire.

Three ways to keep history costs sane

• Window the transcript. Keep the last N turns, not all of them. Most conversations don’t need turn 3 verbatim by turn 30 — a sliding window caps the quadratic growth at a fixed ceiling.
• Summarize old turns. Compress everything older than the window into a short running summary. You trade a little fidelity for a bounded prompt — and the summary itself becomes stable, cacheable prefix.
• Order the prompt for caching. System prompt first, then transcript, then the new message. Anything volatile (timestamps, per-request context) goes after the stable prefix or it busts the cache every turn.

Which one are you actually building?

The tell-tale test from the archetype specs: if you’re including the last five messages in every prompt, you’re running a chatbot with history — even if you think of it as a simple bot — and you should budget like one. Conversely, if conversations are one or two turns, don’t pay the history archetype’s numbers; model it stateless. The two profiles differ by 2× at the averages and far more at the tails, so picking the right one is the single biggest accuracy lever in your chatbot estimate.