Llama 4 Scout currently advertises the longest context window among widely deployed open-weight models, with Meta stating support for up to 10 million tokens, far beyond the 128,000 to 1 million token windows common in other open models. In practice, the advertised maximum and the length at which a model still retains strong accuracy are different numbers, and independent long-context evaluations, often called needle-in-a-haystack tests, tend to show meaningful degradation well before the theoretical ceiling on any model, Llama 4 included. Qwen 2.5's long-context variant supports up to 1 million tokens and has shown solid retrieval accuracy in third-party tests, while Llama 4 Maverick and Gemma 3 sit in the 128,000 to 1 million range depending on configuration. For most enterprise document workloads, such as analyzing a full contract set or a codebase, a well-tested 128,000 to 1 million token window is usually more useful than an unverified multi-million token claim. Serving very long contexts also multiplies KV-cache memory requirements, which directly affects how many GPUs a deployment needs. As an NVIDIA Inception Program member, Nanobase AI sizes GPU memory and validates real retrieval accuracy at the context length a client's documents actually require.
Advertised maximum and effective accuracy are different numbers
A model's marketed context window describes the longest sequence it can technically accept, not the length at which it still retrieves and reasons over information accurately. Independent long-context evaluations, commonly called needle-in-a-haystack tests, place a specific fact at varying depths within a long document and check whether the model retrieves it correctly, and results across the industry consistently show meaningful accuracy degradation well before a model's theoretical ceiling, regardless of which model is tested.
Never plan a deployment around a model's advertised maximum context length; test retrieval accuracy at the length your actual documents require.
Running your own needle-in-a-haystack test
- Build a test document at the length your real use case requires, for example 50,000 or 200,000 tokens.
- Insert a distinct, unambiguous fact at several depths: near the start, middle and end of the document.
- Ask the model a question only answerable using that inserted fact.
- Repeat across multiple depths and multiple documents to check for consistent degradation patterns.
- Compare results across your shortlisted models at the same target length before committing to one.
A model that performs well at 32,000 tokens is not guaranteed to perform well at 200,000 tokens; test at the length you will actually use, not a shorter proxy length.
Long context has a direct GPU memory cost
Serving longer context multiplies KV cache memory requirements linearly with sequence length, which directly affects how many GPUs a deployment needs regardless of how large the advertised context window is. A 70B-class model serving a handful of 4,000-token conversations needs far less KV cache memory than the same model serving even one request at 200,000 tokens, and this cost applies on top of the model's base weight memory.
| Context length | Relative KV cache size | Practical implication |
|---|---|---|
| 4K tokens | Baseline | Minimal impact on concurrent capacity |
| 32K tokens | ~8x baseline | Noticeable reduction in concurrent sessions per GPU |
| 128K tokens | ~32x baseline | Often requires additional GPU memory headroom |
| 1M tokens | ~250x baseline | Typically requires multi-GPU memory pooling for meaningful concurrency |
Every added token of context window used in production consumes GPU memory that could otherwise support more concurrent users; size hardware around your realistic average context length, not the maximum offered.
Techniques that reduce the real cost
Attention variants like grouped-query attention, used across most current open-weight model families, already reduce KV cache size substantially compared to older architectures. Beyond architecture choice, KV cache quantization and prompt caching for repeated system prompts or retrieved documents can meaningfully cut the effective memory cost of long-context serving without reducing the context window itself. These techniques matter more as average request length grows, since their savings scale with the same factor that makes long context expensive in the first place.
KV cache quantization and prompt caching are worth implementing specifically once average context length grows past a few thousand tokens, since the savings compound at scale.
Frequently asked questions
Which model currently has the longest advertised context window?
Llama 4 Scout advertises up to 10 million tokens, far beyond the 128,000 to 1 million token windows common elsewhere. As with any model, verify effective accuracy at the length your use case needs rather than relying on the advertised maximum alone.
Does a longer context window always mean better performance on long documents?
No. Retrieval accuracy at long lengths varies by model and by where in the document the relevant information sits. A shorter but well-validated context window can outperform a longer, unvalidated one on your actual documents.
How much does serving a 1 million token context actually cost?
It depends on model size, precision and concurrency needs, but KV cache at that length is typically hundreds of times larger than at 4,000 tokens, often requiring multi-GPU memory pooling even for a single request at moderate concurrency.
How Nanobase AI helps
As an NVIDIA Inception Program member, Nanobase AI sizes GPU memory and validates real retrieval accuracy at the context length a client's documents actually require, rather than provisioning around an advertised maximum. See our GPU sizing guide for large models or the related question on summarizing long documents. Explore our solutions.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.