Deploying an on-premise large language model at an insurance company starts with sizing the GPU infrastructure to the model and workload, since a 70 billion parameter model needs roughly 140 GB of memory in FP16, about 70 GB in FP8, or around 38 GB in INT4, plus twenty to fifty percent additional headroom for the key-value cache under real concurrent load, which typically points toward one or more NVIDIA H100 GPUs with 80 GB of HBM3 memory and 3.35 TB per second of bandwidth, or H200 GPUs with 141 GB of HBM3e for larger models or higher throughput. The serving layer usually runs on vLLM, TensorRT-LLM, or NVIDIA NIM behind a Kubernetes cluster using the GPU Operator for scheduling, with retrieval augmented generation layered on top to ground answers in policy, claims, and underwriting documents. Because claims and medical data are highly sensitive, the deployment should include role based access control, encryption at rest and in transit, and audit logging sufficient to satisfy GDPR or similar regional requirements from day one rather than added later. Most insurers start with one well scoped use case on a single GPU node before scaling to a larger cluster. Nanobase AI, an NVIDIA Inception Program member, sizes, installs, and operates these on-premise LLM deployments end to end for insurers.
Data sensitivity should decide the deployment zone before hardware does
Insurance data isn't uniformly sensitive, and treating it that way leads to either over-engineering a simple use case or under-protecting a genuinely sensitive one. The first planning decision in an on-prem LLM deployment isn't GPU sizing, it's zoning: which data can touch which environment, since claims and medical data typically require stricter isolation than aggregate policy or marketing data.
A data zoning framework
| Data type | Sensitivity | Typical deployment zone |
|---|---|---|
| Marketing content, general product FAQs | Low | Cloud API acceptable |
| Aggregate policy or pricing data | Medium | Private cloud (VPC) |
| Claims files, medical records, SIU investigation data | High | On-premise or fully isolated private cloud |
| Litigation strategy documents | High | On-premise, restricted access even internally |
Getting this zoning wrong in the permissive direction, by running high-sensitivity data through a public API for convenience, is the mistake that draws the most serious compliance and reputational consequences, so this decision should be made explicit and documented, not left to whichever team builds a given use case first.
Sizing GPUs by use case, not by a single company-wide number
| Use case | Typical model size | Approximate GPU need |
|---|---|---|
| Policyholder chat assistant | 7B–13B parameters | Single mid-range GPU, low concurrency |
| Broker copilot with RAG | 13B–70B parameters | One to a few H100 GPUs depending on concurrency |
| Document extraction (claims, medical) | 7B–34B vision-language or extraction model | One to a few GPUs, throughput-dependent |
| Fraud or risk scoring | Often smaller specialized models, not always LLM-based | Modest GPU or even CPU-viable in some cases |
A 70 billion parameter model needs roughly 140 GB of memory in FP16, about 70 GB in FP8, or around 38 GB in INT4, plus twenty to fifty percent additional headroom for the key-value cache under real concurrent load. That sizing typically points toward one or more NVIDIA H100 GPUs with 80 GB of HBM3 memory and 3.35 TB per second of bandwidth, or H200 GPUs with 141 GB of HBM3e for larger models or higher concurrent throughput.
A phased deployment plan
- Scope one well-defined use case with a clear data zone and a defined user group, rather than a company-wide platform on day one.
- Size and install GPU infrastructure to that use case's actual concurrency and model size needs, typically starting with a single node.
- Stand up the serving layer, commonly vLLM, TensorRT-LLM, or NVIDIA NIM, behind a Kubernetes cluster using the GPU Operator for scheduling.
- Layer retrieval augmented generation on top to ground answers in policy, claims, or underwriting documents relevant to the use case.
- Build in role-based access control, encryption at rest and in transit, and audit logging from day one, since retrofitting these after launch is harder than including them in the initial architecture.
- Scale to additional use cases and a larger cluster only once the first deployment is stable and its access patterns are understood.
Frequently asked questions
Do we need a full GPU cluster to start, or can one node work?
Most insurers start with one well-scoped use case on a single GPU node before scaling to a larger cluster, since starting small validates the use case and access patterns before committing to larger capital infrastructure spending.
Which inference engine should we choose for an insurance deployment?
vLLM, TensorRT-LLM, and NVIDIA NIM are all viable choices, and the right one depends on your existing infrastructure, model format preferences, and team familiarity rather than a universal best answer; NVIDIA NIM often appeals to teams wanting a more managed deployment experience.
How does GDPR or similar regional law affect this architecture?
It generally reinforces the case for on-premise or tightly controlled private cloud deployment for personal and sensitive data, and requires audit logging and access control sufficient to demonstrate compliance on request, which should be built into the architecture rather than added later.
What's the most underestimated cost in these projects?
Ongoing operations, including model updates, retrieval quality tuning, and monitoring, is commonly underestimated relative to the initial hardware and setup cost, so budget a realistic ongoing allocation rather than treating deployment as a one-time project.
How Nanobase AI helps
Nanobase AI, an NVIDIA Inception Program member, sizes, installs, and operates on-premise LLM deployments end to end for insurers, starting with a data zoning assessment before any hardware decision is made. See our on-premise LLM deployment guide and our GPU comparison across H100, H200, and B200 for deeper infrastructure detail, or our solutions for the full deployment service.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.