Banks deploy LLMs on-premise for data security by running open-weight models such as Llama, Qwen, or Mistral entirely within their own data center or a private cloud tenancy they control, so customer and transaction data never crosses the network boundary to a third-party API provider. The typical architecture sizes GPU capacity, often NVIDIA H100 or H200 systems, to the chosen model, serves it through an inference engine like vLLM, TensorRT-LLM, or NVIDIA NIM, and places it behind the bank's existing identity and access controls, network segmentation, and audit logging rather than exposing it externally. A 70B-class model needs roughly 140 GB of memory in FP16 or about 70 GB in FP8, which typically maps to one or two H100 or H200 GPUs depending on quantization and expected concurrent usage. Data governance layers on top restrict which internal documents a retrieval system can surface to which employee roles, since an internal assistant should not casually expose one department's confidential data to another. This approach also satisfies regulators and auditors who expect evidence that sensitive data stayed inside a defined perimeter throughout processing. Nanobase AI, a Silicon Valley enterprise AI engineering company, designs, installs, and operates these on-premise LLM deployments for banks end to end.
Data security starts with a sizing decision, not a firewall rule
Discussions of on-premise LLM security often jump straight to network segmentation and access controls, but the sizing decision made before any hardware is ordered determines whether the deployment can even serve production traffic without a costly redo. Getting GPU capacity right for the chosen model size and concurrency target is the first security decision, not a separate performance concern, because an undersized deployment under real load tends to get "fixed" with shortcuts, such as routing overflow traffic to an external API, that quietly reintroduce the exact data exposure the on-premise architecture was built to avoid.
Sizing by model class
| Model class | Approximate memory need | Typical GPU configuration | Inference engine options |
|---|---|---|---|
| 7B–13B | 15–30 GB (FP16) | Single H100 or smaller GPU | vLLM, TensorRT-LLM |
| 70B | ~140 GB FP16 / ~70 GB FP8 | One to two H100 or H200 GPUs, depending on quantization and concurrency | vLLM, TensorRT-LLM, NVIDIA NIM |
| Larger / high-concurrency deployments | Scales with context length and concurrent users | Multi-GPU or multi-node H100/H200 clusters | TensorRT-LLM, NVIDIA NIM with orchestration |
Quantization choice (FP16 vs. FP8) changes the GPU count needed for the same model by roughly half, which is a meaningful capital decision that should be tested against the bank's actual accuracy tolerance before committing to hardware.
The network isolation pattern that satisfies auditors
A production on-premise deployment sits behind the same identity and access management, network segmentation, and audit logging the bank already applies to its core systems, rather than being treated as a standalone appliance. The model runs on GPU infrastructure with no route to the public internet, inference requests come through an internal API gateway that enforces authentication and logs every call, and a separate data governance layer restricts which internal documents a retrieval system can surface to which employee role. This last point matters because an internal assistant with unrestricted document access can casually expose one department's confidential data to another employee who technically has network access but no legitimate need to see it, a failure mode that has nothing to do with the model itself.
A deployment checklist
- Determine the target model size and quantization level based on accuracy testing against the bank's actual use case, not a generic benchmark.
- Size GPU capacity for peak expected concurrency, including headroom for KV-cache, not just the model weights alone.
- Select an inference engine, vLLM for flexibility, TensorRT-LLM for maximum throughput, or NVIDIA NIM for a packaged deployment, based on the team's operational capacity.
- Place the deployment behind existing identity and access controls with no external network route.
- Build role-based document access controls before connecting any retrieval system to internal content.
- Establish audit logging covering every prompt, retrieved document, and generated output before go-live.
Skipping the role-based document access step is the most common gap in otherwise well-secured on-premise deployments, since network isolation alone does not prevent an over-broad internal assistant from surfacing data across department boundaries.
Frequently asked questions
Is on-premise deployment always necessary for banking LLM use cases?
Not for every use case; lower-sensitivity applications can sometimes use enterprise cloud APIs with strong contractual data protections, but anything touching real customer or transaction data typically needs the stronger guarantee of infrastructure the bank fully controls.
How much GPU capacity does a mid-size bank typically need to start?
It depends heavily on the target model size and expected concurrent users; a focused pilot with one department often runs comfortably on a single high-memory GPU node, with capacity scaled up only once usage patterns from that pilot are known.
Which inference engine is easiest for a bank's infrastructure team to operate?
vLLM is generally considered the most approachable starting point given its simpler operational model, while TensorRT-LLM and NVIDIA NIM offer higher throughput or more packaged deployment respectively at the cost of additional operational complexity.
Does on-premise deployment eliminate the need for EU AI Act compliance work?
No, deployment location affects data security obligations but not AI Act classification; a high-risk use case still requires the same documentation, human oversight, and risk management regardless of whether the model runs on-premise or in the cloud.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, designs, installs, and operates these on-premise LLM deployments for banks end to end, from GPU sizing through role-based access control. See our GPU sizing guide and vLLM vs. TensorRT-LLM comparison for the infrastructure detail behind this sizing table.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.