Most teams treating API costs as fixed are overlooking something specific: prompt token count directly determines a significant portion of their monthly bill. A system prompt that runs 800 tokens on every request costs more than one that achieves the same behavioral output in 300 tokens. The difference is not trivial at scale.
Where token waste typically hides
Verbose role descriptions, redundant instruction repetition, and over-specified output formatting instructions are the three most common sources of bloat. A system prompt telling the model to respond professionally, avoid jargon, use clear language, write in a friendly tone, and keep answers concise is saying one thing five ways. Consolidating to a single precise instruction reduces token count and often improves consistency.
Recent shifts in prompt compression techniques
Structured prompting formats — particularly using XML-style tags to delineate context, instructions, and examples — emerged from Anthropic and several research teams as more token-efficient than prose-heavy prompts. Models trained on instruction-following data respond reliably to tighter formats. Teams switching from paragraph-style system prompts to structured tags reported measurable reductions in input token counts.
Chain-of-thought has a cost tradeoff
Asking models to reason step by step improves accuracy on complex queries but increases output tokens substantially. For straightforward FAQ-style conversational bots, chain-of-thought is overhead. Reserving it for genuinely ambiguous queries — using a classifier to route requests — keeps costs proportional to actual task complexity.
Prompt audits take a few hours and often reveal savings that persist indefinitely.