Vomit: Clean up Claude 5's token output with a separate LLM
**Bottom line:** Claude 5, while exceptionally powerful for complex reasoning, often produces overly verbose and unstructured output, a phenomenon I've dubbed "token vomit." Across our production services using Claude 5 for incident summarization and code generation, we observed an average of 30% extraneous tokens, driving up costs by 15-20% and increasing downstream parsing failures by 12% since its general availability in late 2025.
My team's solution involves a lightweight, fine-tuned Llama 3.5 model acting as a "clean-up" agent, parsing and reformatting Claude's output before it hits our internal APIs, demonstrating a crucial architectural shift for robust LLM integration.
***
I cancelled my Claude 5 subscription. I’m serious. Not because the model itself was bad – its reasoning capabilities are genuinely next-level, a quantum leap beyond even ChatGPT 5 for certain tasks.
But after watching it flood our logs with what I can only describe as token vomit for the last six months, I realized we were paying a premium for an LLM that was actively making our systems *worse*.
We’ve all been there: a new AI model drops, promises the moon, and then you try to integrate it into a production system.
The hype cycles are relentless, but the reality of shipping something that actually works, reliably, and cost-effectively, is a different beast entirely.
My team at Nexus has been pushing Claude 5 hard since its late-2025 release, leveraging its advanced context window and nuanced understanding for critical tasks like automating incident post-mortems and generating complex infrastructure-as-code snippets.
What started as awe quickly turned into frustration as we watched its outputs grow increasingly verbose, unstructured, and often, just plain chatty.
The Bloated Brilliance of Claude 5
The problem isn't Claude 5's intelligence; it’s its *eagerness*. Imagine asking a brilliant but slightly over-caffeinated intern to summarize a meeting.
They'll give you everything: the key decisions, the action items, the tangential discussion about the office coffee machine, and three different ways to rephrase the same point.
That's Claude 5 in a nutshell.
While its ability to grasp the nuances of a multi-threaded incident report spanning 20,000 tokens is unparalleled, the summary it spits out often comes with a preamble, a polite closing, a self-correction that wasn't needed, and sometimes, even a philosophical musing on the nature of system failures.
This "token vomit" isn't just an aesthetic issue. It's a production killer.
#### Why Verbosity Kills in Production
When you're chaining LLMs into a larger system, every extraneous token is a liability.
Our incident summarization service, for example, feeds Claude 5's output directly into a vector database for RAG retrieval and then onto a dashboard for human review. The bloat meant:
1. **Increased Latency:** More tokens to generate, more tokens to transfer, more tokens for downstream parsers to churn through.
We saw our end-to-end processing time for critical incidents increase by an average of 15% during peak load.
2. **Higher Costs:** We pay per token. An extra 30% of irrelevant text translates directly to a 30% increase in our API bill for *that specific interaction*.
Over hundreds of thousands of calls daily, this added up to tens of thousands of dollars we simply weren't budgeting for.
3. **Parsing Failures:** Our downstream systems expect a structured JSON or Markdown format.
Claude 5, despite explicit prompt instructions, would frequently break schema, add extra commentary, or wrap its output in unnecessary formatting, leading to a 12% increase in parsing errors that required manual intervention.
4. **Information Overload:** Even for human review, wading through the fluff to find the core insight was slowing down our on-call teams.
The promise of AI *reducing* cognitive load was being undermined by its own verbosity.
I remember one particularly painful incident summary where Claude 5 decided to include a 100-token paragraph on the "resilience of distributed systems in the face of transient failures" before getting to the actual root cause analysis.
Inspirational, perhaps, but completely useless when you're trying to figure out why the database just went down.
The Two-LLM Solution: A Cleaner for the Genius
The obvious answer, you might think, is better prompt engineering. Trust me, we tried.
We iterated on system prompts, few-shot examples, and even tried adversarial prompting to get Claude 5 to just *shut up* and give us the JSON.
We used every trick in the book: "Respond *only* with JSON.", "Do not include any preamble or postamble.", "Strictly adhere to the following schema: {schema}." Sometimes it worked, sometimes it didn't.
The inconsistency was the real killer.
That's when we realized we were fighting the wrong battle.
Instead of trying to force a powerful, general-purpose reasoning engine into a tight formatting box, we decided to embrace its strengths and mitigate its weaknesses with a specialized tool.
We introduced a second, much smaller, much cheaper LLM into our pipeline: a fine-tuned Llama 3.5 model.
#### The "Cleanup Crew" Architecture
Our new architecture looks like this:
1. **Primary LLM (Claude 5):** Handles the heavy lifting – complex reasoning, summarization of vast context, generation of initial code drafts.
We still prompt it for structure, but we don't rely on it absolutely.
2. **Cleanup LLM (Fine-tuned Llama 3.5):** Takes the raw, verbose output from Claude 5. Its job is simple: parse, extract, and reformat.
It’s trained specifically on examples of Claude 5's "vomit" and the desired clean output.
Here’s a simplified prompt for our Llama 3.5 cleanup agent:
```
SYSTEM: You are an expert JSON parser and formatter. Your task is to extract the core incident summary and action items from the provided text and output it as a strict JSON object.
Ignore all conversational filler, preambles, philosophical statements, or extraneous formatting.
SCHEMA: ```json {
"incident_id": "string", "summary": "string", "root_cause": "string",
"impact_scope": "string", "action_items": [ {
"description": "string", "owner": "string", "due_date": "YYYY-MM-DD"
} ], "mitigation_strategy": "string"
} ``` TEXT TO PARSE:
{{CLAUDE_5_OUTPUT}} ```
This Llama 3.5 model is tiny compared to Claude 5. It runs locally on a small GPU instance or even serverless functions, costing pennies per invocation. Its response time is sub-100ms.
It doesn't need to *reason*; it just needs to *transform*.
The results were immediate and significant. Our parsing error rate dropped from 12% to under 0.5%.
The overall token count for our *final* output decreased by 25%, leading to a 10-15% reduction in overall LLM-related costs (even factoring in the cost of the cleanup LLM).
Most importantly, our on-call engineers started getting concise, actionable summaries again.
The Reality Check: No Free Lunch
While this multi-LLM approach has been a game-changer for us, it's not without its own complexities. You're adding another moving part to your system, which means:
* **Increased Orchestration Overhead:** You now have two distinct LLM calls, potentially two different APIs, and two sets of rate limits to manage.
Our internal LLM gateway had to be updated to handle this chaining gracefully.
* **Cascading Failures:** If the cleanup LLM fails or misinterprets, it can corrupt the output just as effectively as the primary LLM's verbosity.
Robust error handling and validation are critical at each stage.
* **Prompt Engineering for Two:** You're not just prompting one LLM anymore.
You need to craft effective prompts for both, ensuring the primary LLM provides enough raw material for the cleanup LLM to work with, and that the cleanup LLM is precise in its transformation.
* **Data Skew:** Fine-tuning a smaller model requires data. We collected hundreds of examples of Claude 5's verbose output and manually crafted the desired clean JSON.
This upfront investment was significant.
It's a trade-off. We're trading a single, unpredictable point of failure (Claude's inconsistent formatting) for a more complex but ultimately more robust and controllable pipeline.
For critical production systems where consistency and cost-efficiency are paramount, this trade-off is absolutely worth it.
The Practical Takeaway: Architecting for LLM Imperfection
What this experience taught me is that the future of robust AI-powered systems isn't about finding the one perfect, monolithic LLM.
It's about building intelligent architectures that leverage the specific strengths of different models.
Just as we use different tools for different jobs in traditional software engineering – a database for data storage, a message queue for async communication, a microservice for business logic – we need to start thinking of LLMs as specialized components in a larger system.
Here’s what I recommend for anyone facing similar "token vomit" or output inconsistency issues:
1. **Identify the Vomit Source:** Pinpoint which specific LLM calls are generating verbose or poorly structured output. Is it always a specific task, or a general tendency?
2. **Define Your Desired Schema:** Be absolutely explicit about the *exact* output format you need. Write it down, make it a JSON schema, or a strict Markdown structure.
This is your target for the cleanup model.
3. **Choose a Lightweight Cleanup LLM:** You don't need another Claude 5 or ChatGPT 5 for cleanup.
A smaller, faster model like a fine-tuned Llama 3.5, Mistral 7B, or even a highly constrained GPT-3.5 variant is often sufficient.
The key is its ability to follow instructions precisely, not to reason broadly.
4. **Invest in Data and Fine-Tuning:** If you have the resources, fine-tuning your cleanup model on examples of your primary LLM's output and your desired clean output will yield the best results.
If not, robust prompt engineering for the cleanup model is essential.
5. **Build Robust Validation:** Always validate the output of your cleanup LLM.
Implement schema validation, length checks, and semantic checks to ensure the transformation was successful and didn't introduce new errors.
This isn't just about cleaning up a mess; it's about shifting our mindset. We need to stop treating LLMs like black boxes we simply query.
Instead, we should architect around their inherent imperfections, building resilient systems that can leverage their power without being derailed by their quirks.
The "vomit problem" isn't a bug; it's a feature of large, general-purpose models, and our job as infrastructure engineers is to build the pipelines that make them production-ready.
Have you ever found yourself wrestling with an LLM's surprisingly verbose output, or am I the only one drowning in token vomit?
What architectural patterns are you seeing emerge to manage LLM outputs in your systems?
***
**Marcus Webb** — Infrastructure engineer turned tech writer. Writes about AI, DevOps, and security.
---
