Context Window

Context Window is the finite token budget that bounds what an LLM can ‘see’ during a single inference call. All system instructions, conversation history, retrieved documents, tool outputs, and generated text must fit within this limit. Once the window is full, the model cannot reference earlier content unless it is explicitly re-included.

How it works

During inference, the model computes attention across every token in the context. Quadratic attention complexity means memory and compute grow rapidly as context length increases, which is why extending windows requires architectural changes like sliding-window attention, sparse attention, or rotary position embeddings. Providers set a hard token cap per model.

Key facts

• Units: Measured in tokens, not characters. One token averages roughly 4 English characters.
• Range: Early GPT-3 supported 4,096 tokens; current frontier models support 128K to 1M tokens.
• Input vs. output: Total context equals input tokens plus the generated output tokens.
• Cost impact: Larger contexts cost more; pricing is typically linear with token count.

For builders

Managing context is a core engineering concern. Builders must decide what to include, what to truncate, and what to offload to retrieval systems. Prompt caching can reduce the cost of repeating large, stable context blocks across many calls, which is important for long-document or multi-turn workflows.

Sources

• Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762. arxiv.org
• Brown, T., et al. (2020). Language Models are Few-Shot Learners (GPT-3). arXiv:2005.14165. arxiv.org
• Bommasani, R., et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM. arxiv.org
• NIST. (2023). AI Risk Management Framework (AI RMF 1.0). nist.gov
• Stanford HAI. Foundation Models research portal. hai.stanford.edu