Part 1: The basicsChapter 2 of 92
Context windows, explained
Beginner4 min1 code sample
A context window is the maximum amount of text, counted in tokens, that a model can consider in one request. Everything counts against it: the instructions, the conversation so far, every document you attach, every tool result, and the reply the model is about to write. For a chatbot this is a background detail. For an agent, which collects tool results turn after turn, the context window is the central constraint, and managing it is most of the job.
What fills the window#
In a typical agent run the window holds, roughly in this order:
- The system prompt: role, rules and what "done" looks like.
- Tool definitions: every tool's name, description and input schema. These are sent on every request, so fifty verbose tools are a permanent tax.
- The conversation: the user's request, the model's earlier messages, and the model's reasoning if you keep it.
- Tool results: file contents, search hits, API responses. These are usually the biggest and fastest-growing part.
- Room reserved for the reply (the output limit you set).
Because each turn resends everything so far, a run's total cost grows faster than its length: ten steps read the early messages ten times. That is why caching (prompt caching) and trimming (context engineering) pay off so quickly.
Big windows are not magic#
Windows of several hundred thousand tokens, and in some models a million or more, let an agent hold a whole codebase or a long contract at once. Three cautions apply:
- Quality is not uniform. Models tend to use the start and end of a long context better than the middle, and they can miss a detail buried in a huge pile. Test with your own data rather than trusting a headline number.
- Cost and latency scale with length. Processing a large prompt takes noticeable time before the first output token, and you pay for every token each time unless it is cached.
- More context can mean worse answers. Irrelevant text is a distraction. A tight, relevant 5,000 tokens often beats 200,000 tokens of everything.
Strategies when the window fills up#
| Strategy | How it works | Good for |
|---|---|---|
| Retrieve instead of load | Search a collection and put only the best passages in context (RAG) | Large knowledge bases |
| Summarize old turns | Replace early messages with a short summary | Long conversations |
| Clear old tool results | Drop bulky results the agent has already used | Research and coding runs |
| Write notes to disk | The agent saves findings to a file and re-reads only what it needs | Tasks that outlive one window |
| Split the work | Give a subagent its own fresh window and take back a short report (many agents) | Parallel or very large jobs |
| Choose a bigger window | Switch to a model with more room | When the others cost more than they save |
The foundations chapter on context and memory builds the first four of these in code.
Counting before you send#
Do not guess whether a request will fit. Most providers offer a way to count tokens without running the model, and your agent can use it to decide when to compact. A rough rule for English text is about four characters, or three quarters of a word, per token, but code, other languages and unusual symbols use more. Claude, OpenAI and Gemini each have a counting call; the OpenAI-compatible APIs that serve Llama and local models, and the AI SDK, do not, so those tabs ask for a single output token and read the prompt count from the usage.
src/count.ts
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
const MODEL = "claude-opus-5-5";
const count = await client.messages.countTokens({
model: MODEL,
messages: [{ role: "user", content: "Summarize the attached report in five bullet points." }],
});
console.log(count.input_tokens); // what this request would cost on the way insrc/count.ts
import OpenAI from "openai";
const client = new OpenAI(); // reads OPENAI_API_KEY from the environment
const MODEL = "gpt-6.1-sol";
const count = await client.responses.inputTokens.count({
model: MODEL,
input: "Summarize the attached report in five bullet points.",
});
console.log(count.input_tokens); // what this request would cost on the way insrc/count.ts
import { GoogleGenAI } from "@google/genai";
const client = new GoogleGenAI({}); // reads GEMINI_API_KEY from the environment
const MODEL = "gemini-3.8-flash";
const count = await client.models.countTokens({
model: MODEL,
contents: "Summarize the attached report in five bullet points.",
});
console.log(count.totalTokens); // what this request would cost on the way insrc/count.ts
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://api.together.ai/v1", apiKey: process.env.TOGETHER_API_KEY }); // Together AI; Groq and Fireworks work the same way
const MODEL = "openai/gpt-oss-120b";
// The OpenAI-compatible API has no counting endpoint, so ask for a single output token and read the usage:
// the prompt is counted exactly as the model will see it (chat template included) for the price of one token.
// gpt-oss is a reasoning model whose thinking shares the max_tokens budget on Together, so the reply text will be empty here; the usage is what you want.
const probe = await client.chat.completions.create({
model: MODEL,
max_tokens: 1,
messages: [{ role: "user", content: "Summarize the attached report in five bullet points." }],
});
console.log(probe.usage?.prompt_tokens); // what this request would cost on the way insrc/count.ts
import { ChatAnthropic } from "@langchain/anthropic"; // or ChatOpenAI, ChatGoogleGenerativeAI, ChatOllama: the rest of the code is identical
const model = new ChatAnthropic({ model: "claude-opus-5-5" }); // reads ANTHROPIC_API_KEY from the environment
// getNumTokens is a local estimate (a general-purpose tokenizer), not the provider's own count: fine for deciding when to compact.
// For an exact figure, call the provider's counting endpoint, as in the other tabs.
const tokens = await model.getNumTokens("Summarize the attached report in five bullet points.");
console.log(tokens); // roughly what this request would cost on the way insrc/count.ts
import OpenAI from "openai";
// A model running on your own machine: Ollama on port 11434, a vLLM server (vllm serve <model>) on port 8000.
const client = new OpenAI({ baseURL: "http://localhost:11434/v1", apiKey: "ollama" }); // the key is ignored locally but must be non-empty
const MODEL = "llama3.3"; // first run: ollama pull llama3.3
// The OpenAI-compatible API has no counting endpoint (vLLM also has a /tokenize endpoint on its own port; Ollama's API has no tokenizing endpoint),
// so ask for a single output token and read the usage. The prompt is counted exactly as the model will see it.
const probe = await client.chat.completions.create({
model: MODEL,
max_tokens: 1,
messages: [{ role: "user", content: "Summarize the attached report in five bullet points." }],
});
console.log(probe.usage?.prompt_tokens); // what this request would cost on the way insrc/count.ts
import { generateText } from "ai";
import { anthropic } from "@ai-sdk/anthropic"; // swap for openai(...) from @ai-sdk/openai or google(...) from @ai-sdk/google
// The AI SDK has no counting call, because counting is a provider feature. Ask for one output token and read the usage instead.
const probe = await generateText({
model: anthropic("claude-opus-5-5"), // reads ANTHROPIC_API_KEY from the environment
maxOutputTokens: 1,
prompt: "Summarize the attached report in five bullet points.",
});
console.log(probe.usage.inputTokens); // what this request would cost on the way inWhere to go next#
If an agent needs information that does not fit in any window, the answer is to fetch it on demand. The next two chapters explain the two ways to adapt a model to your needs, fine-tuning versus prompting, and the most common way to feed it knowledge, retrieval.
Finished this chapter?
Your progress is saved in this browser.
Up next: 3. Fine-tuning vs prompting · 3 min
About this chapter
This unlocks75. Lab 1: Build your first agent loop
On the learning pathsAI from zeroBuild AI agents
Checked against @anthropic-ai/sdk 0.135, @anthropic-ai/claude-agent-sdk 0.3, @openai/agents 0.20, @modelcontextprotocol/server 2.3, @modelcontextprotocol/client 2.3, zod 4, openai 7.45, @google/genai 2.31, ai 7.0, @ai-sdk/anthropic 4.0, @ai-sdk/openai 4.0, @ai-sdk/google 4.0, langchain 1.5, @langchain/core 1.3, @langchain/langgraph 1.5, @langchain/anthropic 1.6, @langchain/openai 1.7, @langchain/google-genai 2.3, @langchain/ollama 1.3, ollama 0.6 on 2026-10-27. Agent tooling changes quickly: where a provider's documentation disagrees with this page, trust the documentation, and tell us on the corrections page.