AI Agent Optimization: A 7-Point Token Reduction Checklist

AI agent optimization often starts with a surprise: your bill. If you are building agents that call large language models (LLMs) repeatedly, you are likely paying for context you do not need. This guide provides a seven-point checklist for token reduction, based on a production field report where a team reduced input tokens from 17,000 to near zero before the first meaningful read. You will learn what worked, what failed, and how to measure the impact yourself.

The Problem: Redundant Context and Hidden Costs

In a typical production agent, every model call starts from zero. The agent must resend everything it "knows": system instructions, tool declarations, conversation history, and past tool results. A recent field report documented a Business Intelligence (BI) agent that re-read 17,000 tokens of instructions and tool manuals before answering a single question. In an offline replay of a three-question conversation, ten model calls consumed 286,522 input tokens before any optimization.

This constant resending of context creates significant overhead. It's not just a cost issue; it also impacts the quality of the agent's responses. When an agent is forced to process a large volume of irrelevant information, it can dilute the signal of the actual query, potentially leading to less accurate or slower answers. Every unnecessary token incurs a cost, both in terms of the LLM invoice and the potential degradation of the agent's performance.

1. Measure Before You Cut

You cannot optimize what you cannot see. The first step in any AI agent optimization strategy is building a benchmark (bench). A field report on production agents detailed how a team built a bench in bench/bi_tokens with five scenarios: creating a dashboard, asking a question, chatting, a three-question conversation, and a forecast.

The bench used an offline replay with a scripted model. This ensures that each change is compared against identical calls, allowing for precise counting of input tokens per call. Crucially, the bench included quality checks. A cut that saves tokens but fails a quality check—such as missing a fact or hallucinating data—is rejected.

Why this matters: If you cut context without quality checks, you risk breaking the agent's reasoning. The field report notes that their best-looking cut initially would have quietly broken the answers. Without a bench, you might reduce costs by a significant percentage and inadvertently lose a substantial portion of your accuracy, never knowing which trade-off occurred.

2. Remove Unused Tools

The cheapest win is often the one you ignore. The BI agent described in the field report had inherited generic chat tools from its platform. These tools were declared in the prompt, adding to the token count, but the agent never called them.

The Technique: Audit your tool declarations. If a tool is not called in your benchmark scenarios, remove it from the prompt sent to the LLM. This means conditionally including tools based on the agent's current task or the user's intent.

The Result: This cut reduced input tokens by 10–15% per scenario on the first call. In real runs, the reduction was 18.5%. No quality regressions were observed because the tools were never actually used.

Code Pattern:

1// Illustrative example: Filter tools based on usage
2function getActiveTools(allTools, usedToolIds) {
3  // In a real application, 'usedToolIds' would be determined dynamically
4  // based on the agent's current task or a pre-analysis of common workflows.
5  return allTools.filter(tool => usedToolIds.includes(tool.id));
6}
7
8// In your agent initialization or turn processing logic
9const platformTools = [
10  { id: 'query_db', description: 'Query the database' },
11  { id: 'generate_chart', description: 'Create a chart' },
12  { id: 'send_email', description: 'Send an email' }, // Often unused in BI agents
13  { id: 'search_web', description: 'Search the web' } // Often unused in BI agents
14];
15
16// For a BI agent focused on data, these might be the only active tools
17const activeToolIdsForBI = ['query_db', 'generate_chart'];
18const activeTools = getActiveTools(platformTools, activeToolIdsForBI);
19
20// The prompt should only include declarations for 'activeTools'
21const prompt = buildSystemPrompt(instructions, activeTools);

3. Shorten Old Tool Results

Agents often carry the full payload of past tool results in their history. If an agent pulled a large table in turn one, that entire table is resent in turns two, three, and four, even if only a summary is needed. This is a primary driver of token bloat in multi-turn conversations.

The Technique: Compress or truncate tool results from past turns. Keep the full data for the current turn. The goal is to provide enough context for the model to remember what happened, without resending every detail.

The Result: Shortening old tool results saved 21–26% on conversations in the reported benchmark. However, the field report warns: never touch the current turn. When the team tried shortening the current turn's context, quality broke in four test scenarios. The model needs the full, unadulterated context of what it is currently working on to make accurate decisions.

Code Pattern:

1# Illustrative example: Truncate old results
2SUMMARY_LENGTH = 150 # Determine this length through rigorous testing with your benchmark
3
4def format_history(messages):
5    formatted = []
6    # Iterate through messages, keeping the last two (current turn's tool output and user prompt) intact
7    # Assuming a turn consists of a user message and potentially a tool output
8    for i, msg in enumerate(messages):
9        # Apply truncation to tool results from previous turns
10        # The exact index for "previous turns" might vary based on your message structure
11        if msg['role'] == 'tool' and i < len(messages) - 2:
12            # Truncate content if it exceeds the summary length
13            content = msg['content'][:SUMMARY_LENGTH] + "..." if len(msg['content']) > SUMMARY_LENGTH else msg['content']
14        else:
15            content = msg['content']
16        formatted.append({
17            'role': msg['role'],
18            'content': content
19        })
20    return formatted

4. Describe Big Payloads Instead of Pasting

Large binary payloads, such as Base64-encoded images, large CSVs, or extensive JSON objects, can consume thousands of tokens. If an agent processes a page image, for example, that Base64 string is often resent on every subsequent call, even if the agent only needs to know that an image was processed.

The Technique: Replace these large payloads in the prompt's history with a concise description or a reference. For example, instead of sending the full Base64 string of an image, send "Attached: screenshot of dashboard v2 (ID: dashboard_v2_img_001)" and ensure the agent can retrieve the full image via a tool if it genuinely needs to re-examine the details. This shifts the burden from constant context to on-demand retrieval.

The Result: By replacing the full payload with a descriptive reference, the large data was no longer re-sent at every turn. The agent retained the ability to access the full data through tools, maintaining functionality while drastically reducing the static context size. This is particularly effective for agents that process visual information or large data extracts.

5. Make Prompts Cache-Friendly

Most LLM providers offer prompt caching, which can significantly reduce the cost of repeated prompt prefixes. However, caching only works if the prefix is identical between calls. If your prompt includes dynamic elements that change on every call—like a timestamp, a randomly ordered list of available tables, or a unique session ID—the cache misses every time, negating any potential savings.

The Technique: Order your prompt components deterministically. Place static instructions first, followed by static tool definitions, and then dynamic history. Crucially, ensure that any lists or schemas (like database table names or column definitions) are sorted consistently, for example, alphabetically by name. This creates a stable, reusable prefix for the LLM.

The Result: In the field report's benchmark, ordering schema tables deterministically raised the reusable prefix of a SQL prompt from 11% to 98%. This meant that nearly all of the prompt's static components were cached, significantly reducing the effective cost of input tokens. Keep in mind that the actual cache hit rate depends on your provider’s caching implementation and how frequently the dynamic parts of your prompt change.

Code Pattern:

1// Illustrative example: Deterministic prompt building
2function buildCachedPrompt(staticInstructions, toolSchemas, dynamicContext) {
3  // Ensure tool schemas are sorted by a stable identifier (e.g., 'id' or 'name')
4  const sortedTools = [...toolSchemas].sort((a, b) => a.id.localeCompare(b.id));
5
6  // Example of a structured, deterministic prompt
7  return [
8    staticInstructions,
9    "--- AVAILABLE TOOLS ---",
10    JSON.stringify(sortedTools, null, 2), // Pretty-print for readability, but ensure consistent formatting
11    "--- CONTEXT AND HISTORY ---",
12    dynamicContext
13  ].join("\n\n");
14}
15
16// Example usage:
17const instructions = "You are a helpful assistant...";
18const toolDefs = [
19  { id: 'tool_b', description: 'Tool B' },
20  { id: 'tool_a', description: 'Tool A' }
21];
22const context = "User asked about sales data.";
23
24const prompt = buildCachedPrompt(instructions, toolDefs, context);
25console.log(prompt);

6. Reduce Round Trips

The biggest real gain often comes from fewer model calls, not just smaller ones. If an agent needs to make five separate calls to the LLM to answer a single user question—perhaps querying, then formatting, then error-checking, then re-querying—you are paying for the overhead of each call five times. Each round trip incurs its own prompt, response, and latency.

The Technique: Optimize the agent’s logic to batch requests or consolidate steps into fewer, more complex interactions. For example, instead of a multi-step process, can you design a single, more capable tool or a more comprehensive prompt that allows the model to achieve the outcome in one or two passes? This might involve giving the model more information upfront or designing tools that can handle more complex operations.

The Result: In a real run documented in the field report, reducing round trips led to a 41% reduction in tokens for a specific question because the model was able to ask once instead of five times. This approach only pays off if the model can reliably handle the increased complexity in a single pass; over-complicating a single turn can lead to errors or longer processing times.

7. Trim Instructions

This is typically the last step in AI agent optimization because it often yields the smallest token gains. The field report found that trimming instructions saved about 2% of tokens. While seemingly minor, these savings compound over many calls.

The Technique: Review your system prompt for verbosity. Remove redundant explanations, unnecessary pleasantries, or information that is already implicitly covered by tool definitions or the agent's role. Use concise, imperative language. Focus on clarity and essential constraints.

The Result: A small but consistent reduction in tokens. This step carries the highest risk of confusing the model if you remove necessary nuance or critical operational instructions, so proceed with caution and rely heavily on your quality checks.

Pitfalls and When Not to Use This

Your Bench Will Lie to You a Little. The field report found that their offline bench overestimated real gains by 9–16% and underestimated one significantly. It also missed a failure entirely. Always validate your optimizations with real-world runs and actual cost data from your LLM provider's usage reports.

Quality is Non-Negotiable. If a token reduction breaks a core feature, introduces hallucinations, or degrades the accuracy of your agent's responses, it is not a savings. It is a bug. Your quality checks in the benchmark are crucial for catching these regressions.

When Not to Optimize: If your agent has a very short context window (e.g., under 10 turns for typical interactions) and very low usage volume, the engineering effort required for these optimizations may not justify the marginal cost savings. Focus on optimization when you are operating at scale, have high-volume multi-turn conversations, or are facing significant LLM costs.

Key Takeaways

• Measure first: Build a robust benchmark with quality checks before changing any code.
• Remove unused tools: This is often the cheapest win, potentially saving 10–15% of tokens.
• Compress history, not current turn: Shorten old tool results, but always keep the current turn's context intact.
• Use caching: Order prompts deterministically to maximize cache hits and reusable prefixes.
• Reduce calls: Fewer model calls often yield bigger savings than just smaller prompts.

FAQ

Does trimming instructions hurt model performance? It can. The field report found a 2% token reduction but warns that removing necessary nuance can break quality. Always test thoroughly with your specific scenarios and quality metrics.

How do I know if my prompt is cache-friendly? Check your LLM provider’s dashboard or API responses for cache hit rates. If the hit rate is low, review your prompt construction to ensure the prefix is as static and deterministic as possible.

Can I use this for single-turn agents? Yes, but the gains will generally be smaller. The most significant savings from these techniques come from multi-turn conversations where context history accumulates over time.

What is the "MCP tax"? This term refers to the inherent token overhead in an agent's operation, where system instructions, tool declarations, and conversation history are repeatedly sent to the LLM on every call. It's the cost incurred for context that the agent "remembers" but doesn't actively use in the current turn.