Yes, AI agents can run fully on-premise using open-weight models such as Llama, Qwen or Mistral variants served through vLLM, TensorRT-LLM or NVIDIA NIM on your own GPU infrastructure, with no data leaving your network. The agent orchestration layer, whether LangGraph, the Claude Agent SDK adapted to a local endpoint, or a custom loop, runs the same way regardless of whether the underlying model is a hosted frontier API or a self-hosted open-weight model, since most frameworks talk to models through an OpenAI-compatible interface. The trade-off is capability: open-weight models in the 70B to 120B range handle tool calling and multi-step reasoning well for well-scoped enterprise tasks, but frontier proprietary models still lead on the hardest planning and reasoning benchmarks, so some organizations run a hybrid setup with sensitive data on-premise and only anonymized or non-sensitive reasoning routed to a frontier API. Sizing matters too, since agent workloads with long tool-calling chains benefit from the larger KV-cache headroom that H100 or H200 GPUs provide. Nanobase AI deploys these fully on-premise agent stacks for insurance, finance and government clients whose data residency and compliance requirements rule out cloud-hosted models entirely.
Agent workloads size differently than chat workloads
Sizing GPUs for a chat application and sizing them for an agent running the same model are two different exercises, because an agent's context grows with every tool call, observation and reasoning step within a single task, often far exceeding what a conversational chat turn accumulates. A workflow that calls five tools sequentially, each returning a paragraph of data plus the model's reasoning between calls, can push a single task's context into the tens of thousands of tokens before it finishes, which directly consumes KV-cache memory on the GPU serving it. Sizing for agents means budgeting extra KV-cache headroom for long, multi-step tool-calling chains, not just for the model's static weight footprint.
Matching hardware to a self-hosted agent's model size
| GPU | Memory / bandwidth | Fits well for |
|---|---|---|
| L40S | 48 GB GDDR6 | Smaller open-weight models (7B-13B) for narrow, high-volume agent tasks |
| A100 80 GB | 80 GB HBM2e, 2 TB/s | Mid-size models (30B-70B) with moderate concurrent sessions |
| H100 80 GB | 80 GB HBM3, 3.35 TB/s, 700 W | 70B-class agents with real KV-cache headroom for multi-step tool chains |
| H200 | 141 GB HBM3e, 4.8 TB/s | High-concurrency agent deployments or larger models with long context needs |
| RTX PRO 6000 | 96 GB GDDR7 | Cost-efficient single-node deployments for mid-size agent models |
A 70B model's weights alone need roughly 140 GB in FP16, about 70 GB in FP8, or around 38 GB in INT4, and an agent deployment on top of that typically needs an additional 20 to 50 percent of memory as KV-cache headroom, with the higher end of that range appropriate for workflows involving many sequential tool calls per task. This is why a 70B agent serving concurrent users comfortably often calls for two H100s or a single H200 rather than the single-GPU sizing that a similarly-sized chat deployment might get away with.
The hybrid pattern most enterprises actually run
Fully on-premise agents are technically straightforward once the model is served through vLLM, TensorRT-LLM or NVIDIA NIM behind an OpenAI-compatible interface, since the orchestration layer, whether LangGraph, an SDK-based loop, or a custom implementation, talks to that endpoint the same way it would talk to a hosted API. What varies is capability: open-weight models in the 70B to 120B range now handle tool calling and multi-step reasoning well for well-scoped enterprise tasks, but the very hardest planning and reasoning tasks still tend to favor frontier proprietary models. Most regulated enterprises land on a hybrid pattern, keeping sensitive data and routine tool-calling steps on-premise while routing only the occasional hard reasoning step, stripped of sensitive content, to a frontier API.
A sizing checklist before procurement
Sizing off the model's context window alone, without accounting for realistic tool-call chains, is the single most common way agent deployments under-provision GPU memory.
- Estimate average and worst-case tool-call chain length per task, since that drives per-session context size more than the base prompt.
- Estimate peak concurrent agent sessions, not just total daily volume, since GPU memory is consumed by sessions in flight.
- Choose a quantization level for the served model, trading some accuracy for a smaller weight footprint and more available KV-cache headroom.
- Add the 20 to 50 percent KV-cache headroom guidance on top of the base weight memory for your chosen quantization level.
- Validate the sizing against a realistic load test using your actual tool-calling traces before finalizing hardware, since synthetic benchmarks understate agent-specific memory pressure.
Frequently asked questions
Can agents run on smaller GPUs like an L40S?
Yes, for smaller open-weight models in the 7B to 13B range serving narrow, well-scoped tasks with modest concurrency, an L40S's 48 GB is often sufficient. Larger models or high-concurrency agent workloads with long tool-calling chains outgrow that memory budget quickly.
Does an on-premise agent lose capability compared to a frontier API?
On the hardest, most ambiguous planning tasks, yes, current open-weight models in realistic serving sizes generally trail top frontier models. On well-scoped enterprise tasks such as document extraction, classification and routine multi-step workflows, the gap is much smaller and often not decisive.
How much does tool-calling chain length actually affect memory needs?
Significantly. A task with two tool calls and a task with fifteen sequential tool calls can differ by an order of magnitude in accumulated context per session, which is why sizing on the base model's context window alone, without accounting for realistic agent chain lengths, routinely under-provisions memory.
Is InfiniBand or high-speed networking needed for a single-node agent deployment?
Not for a single-node deployment serving one model. Multi-node setups, whether for a larger model requiring tensor parallelism or for higher aggregate throughput across GPUs, benefit from InfiniBand to avoid networking becoming the bottleneck instead of the GPUs themselves.
How Nanobase AI helps
Nanobase AI, an accepted member of the NVIDIA Inception Program, sizes and deploys these fully on-premise agent stacks for insurance, finance and government clients whose data residency and compliance requirements rule out cloud-hosted models entirely. The underlying GPU sizing methodology is covered further in how many GPUs for 70B and 405B models, and hardware selection specifics in H100 vs H200 vs B200 for LLM inference.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.