LLM context window: Tokens, limits, and what to do about them

Say you're reviewing a vendor contract. It’s long, so you upload the whole thing to Claude or ChatGPT, along with last year's contract for comparison and a long email thread about the negotiation. Then you ask: "Did the auto-renewal terms change?"

The model answers confidently: no, the auto-renewal clause is the same in both versions. Except it isn't. The right answer was in the window the whole time, yet the model couldn't find it in the pile.

Now, scale that failure to an enterprise running AI agents, where every oversized prompt costs money. That's thousands of tokens spent, per question, on answers you can't trust.

Both problems trace back to one thing: the context window. The more strategic you are about what goes into it, the better your outputs get. So the real question isn't how much data you can fit inside, or whether bigger is inherently better. It's how to optimize it. Here are the architectural decisions you need to make to get there.

What does the LLM context window mean?

Think of the context window as the LLM’s working memory. It's the total amount of text a model can hold in a single interaction. Everything counts: your prompt, the documents you upload, the conversation history, and the model's response. All of it sits inside that working memory at once, and it has a limit. 

That limit determines how long a conversation the model can carry before it starts losing track of details from earlier exchanges. Once the conversation outgrows the window, the earliest parts fall away, and the model effectively forgets them.

Understanding AI tokens 

AI tokens are the unit a context window is measured in. A token is the basic unit of text an LLM reads and produces. Before a model can process a sentence, it breaks that sentence into tokens: short fragments of characters, words, or punctuation.

Example of tokens for GPT-5.x and O1/3 models (source)

Tokens are also how most AI platforms charge you. In this case, prices are based on tokens consumed and generated, with separate rates for input and output. Essentially, you pay to send context in and pay again for the response the model generates.

Databricks' Genie is a perfect example of how this plays out in an analytics workflow. When you ask a question, the platform translates it into SQL, runs it against the data, and returns an answer. Every step in this chain consumes tokens: the natural language input, the schema context, and the generated SQL. For platforms serving hundreds of concurrent users, that volume compounds fast, which is why most enforce per-minute token limits to keep costs from spiraling.

Why bigger LLM context windows aren’t the solution

Here's the twist: even if the context window had an infinite limit and you could feed in everything, these problems would still remain:

Token caps

With models now offering million-token context windows, you will still eventually hit a wall. Take a standard enterprise warehouse: over 1,000 tables, plus all the schemas, comments, glossaries, and lineage data that come with them. You simply cannot fit all of that into a single prompt.

Truncation looks like the obvious fix, but it creates a blind spot. The context you cut to save space might be the exact table the model needs to answer accurately, and you usually don't find out until the answer is already wrong.

Context rot 

Say you get past the size limit and fit everything into the window. You'd think more context means better answers. It's often the opposite. As the input grows longer, the model gets worse at holding onto what matters, so it fixates on the wrong things and repeats mistakes. That's context rot. 

Chroma tested 18 models, including current frontiers like Claude, and found that performance gets less reliable as the input gets longer. Accuracy can plummet below 30%, a reality Anthropic’s data team experienced firsthand. 

And in a real business, context grows fast. New definitions, new reporting rules, new data sources — they all end up in the same window the model is already struggling to reason across. Every addition pushes accuracy further down the curve.

Look at semiconductor giant Arm. Their procurement team manages thousands of suppliers, contracts, and spend categories across dozens of business units. Every one of those pieces carries its own definitions, thresholds, and reporting rules. Cramming all of that into a single context window doesn't help the model reason better.

The "Lost in the middle" phenomenon

Simply stuffing a model to its limit does not guarantee perfect recall. Research indicates that LLMs are highly adept at processing information at the beginning and end of their context, but frequently miss data buried in the middle—a pattern known as "lost in the middle."

For conversational analytics, this is a massive liability. You might successfully cram 800 table definitions into a single prompt. But if the five tables you actually need sit in the middle, that is exactly where the model pays the least attention. Just because information fits in the window doesn't mean the AI will use it.

Cost and latency

Each token incurs costs, and every long call increases latency. In agentic analytics, a single question can trigger a whole chain reaction of tool calls, and every single one adds to the bill. That's why engineering teams are careful with long prompts. A giant context window isn't free real estate — it's a recurring tax on both your cloud budget and your users' patience.

What happens when context isn't managed

AI context is everything the model draws on to interpret a question: the schemas, definitions, business logic, and history. For smaller tasks like summarization, there's little to manage, so a good prompt carries most of the weight. Agentic systems are different. They run multi-step analysis, and every step leans on enterprise context to stay accurate.

The catch is that context is not static. It changes as your business evolves, and when no one maintains it, it degrades in three ways:

Documentation becomes stale

Static documentation decays the second an engineer ships a migration. If someone renames a column, deprecates a dimension, or changes a join key, your prompts will keep running. They just quietly return the wrong numbers. You won't even realize the system is broken until a stakeholder flags the bad data three weeks later.

Definitions conflict across teams

Different teams use the same business terms to mean different things. Even something as basic as "product" splits three ways — is it a SKU, a subscription tier, or a feature bundle? Ask about "customers" or "revenue" and the confusion only compounds. Traditional semantic layers try to formalize these definitions, but they still struggle with the nuance. The model has to know which definition applies to which user in which scenario, or the same question produces three different answers and nobody trusts any of them.

Logic only lives in people's minds

The most important business logic often never makes it into the schema. Analysts know which tables are trustworthy, which columns are outdated, and which test records to filter out. A column called is_active, for example, may technically hold true for thousands of users — but internally, the analytics team defines an active user as someone with:

• last_login within the past 30 days

• transaction_count greater than zero

• a paid subscription enabled

None of that logic lives in the database definition. It lives in Slack threads, analyst notebooks, half-updated dashboards, and the head of whoever built the original model. When those analysts leave, the knowledge walks out with them, and the next person to query is_active gets a number that looks right but isn’t. 

Effective methods for managing context

LLMs fail in two directions: give them too little context, and they improvise by inventing columns, hallucinating joins, guessing at definitions they were never given. Give them too much, and they lose the thread of the conversation or time out before finishing the job.

Navigating this narrow window requires context engineering: the discipline of controlling exactly what a model sees at every step. It's knowing what to include, what to leave out, what to fetch dynamically, and what to bake in permanently.

Here are the core methods teams use to strike that balance:

Let's look at each approach in detail:

In-context learning and prompt engineering

The simplest approach is direct prompting: you load schema definitions and a few example SQL queries directly into the context window.

The model studies the examples and attempts to generate similar patterns in the output. This approach works surprisingly well for small databases and requires almost no infrastructure. However, as schemas grow larger, the prompts become unstable. Ordering matters, positional bias creeps in, and token usage explodes. What works flawlessly for a 20-table demo will collapse against a real enterprise warehouse.

Retrieval-augmented generation (RAG)

RAG is a flexible approach that supplies an LLM with precise context on the fly. Instead of loading an entire schema into every prompt, you connect the model to an external knowledge base, like your documentation or semantic layer.

The mechanics are straightforward: the system passes relevant metadata to an embedding model, which converts it into vector representations to capture its semantic meaning. Those vectors are stored in a vector database alongside their original text, so when you ask a question, the system retrieves only the most relevant pieces via semantic similarity search and feeds them into the LLM's context window.

This approach keeps the context window manageable, even for massive enterprise warehouses. It also simplifies maintenance, as you only need to refresh the metadata that actually changes. By grounding the LLM's reasoning in retrieved, up-to-date facts rather than its static memory, RAG makes answers to domain-specific questions more accurate.

There is one major caveat: embeddings capture semantic similarity, not business meaning. A search for "customer churn" might pull "customer complaints" just because the concepts are loosely related. To solve this, you can't rely on vectors alone. You need a governed source of context that explicitly maps raw terms to the underlying business logic they refer to.

Fine-tuning

Fine-tuning trains the LLM on your specific natural-language-to-SQL examples. By teaching the model your company-specific syntax upfront, prompts get shorter and inference costs drop. It's a good fit for:

• SQL dialect conventions

• Repeated query structures

• Stable formatting patterns

• Common templates

But here’s the catch: enterprise warehouses aren't static. Retraining a model for every minor schema change is slow, expensive, and operationally brittle. Even when the schema does hold steady, fine-tuning only teaches the model how your SQL looks.  It doesn't teach it what the business terms mean or how to interpret them.

Agentic routing

Agentic routing breaks a complex question into a sequence of small, specialized LLM calls. A central "planner" dictates which agent handles which step, passing only the strictly relevant context to each one.

Think of a well-run analytics team. You don’t ask one analyst to interpret the prompt, memorize the entire warehouse, write the SQL, and check it for errors. You split the work: one scopes the question, another maps the tables, another writes the SQL. Agentic routing applies this exact division of labor to LLM workflows.

By giving each agent a tightly scoped context, you reduce positional bias and keep prompts manageable. However, this introduces orchestration complexity. Every sub-agent relies on high-quality upstream data. If the planner misinterprets a business term, that error cascades. And because validation agents typically catch structural SQL errors rather than semantic ones, a query built on bad context will still pass cleanly through the system.

How does WisdomAI keep enterprise context accurate?

Look at the four methods again. They all assume the same thing: that the enterprise context they're pulling from is accurate, current, and unambiguous. Take that away, and every single one of them fails.

If the schemas are stale, the definitions conflict, and the real logic lives in someone's head, the retrieval can be perfect, but the answers are still wrong.

WisdomAI's Adaptive Context Engine closes that gap. It manages all four layers of enterprise context — business, semantic, governance, and operational — and keeps them current as the business changes. 

• Token costs stay low: Context is grounded in your business definitions, so the model gets answers right on the first attempt instead of burning tokens on wrong guesses and regenerated queries.

• Retrieval is structured: Every question routes to the right authoritative source before any query is written. Forget semantic-similarity guesswork. The context travels with the question.

• Stale context gets caught before it ships: Drift Detection flags schema and definition changes the moment upstream sources shift. Outdated context never enters a prompt.

• Ambiguity is resolved at the source: When "product" means three different things across teams, the engine routes the clash for human review. Only the correct definition enters the window.

• Every answer is auditable: The evidence trail shows exactly which definitions and join paths entered the window. When a number doesn't match the table, the trace is there.

The real headline isn't that you need a longer context window. That approach is essentially a brute-force hack. What matters is structured retrieval and efficient context management. And an engine to run that mechanism is the differentiator.

Schedule a demo to see the difference WisdomAI’s Adaptive Context Engine has made for enterprises like Cisco and ConocoPhillips achieve over 95% accurate AI analytics.