What is the cost per user of a self-hosted LLM assistant?

The cost per user of a self-hosted LLM assistant is calculated by dividing the total monthly infrastructure cost, including amortized GPU hardware or cloud rental, electricity, and a share of operations time, by the number of active users, and it drops sharply as more users share the same GPU capacity since inference infrastructure has largely fixed costs up to a given concurrency ceiling. A single GPU or small cluster sized for a target concurrency can often support anywhere from dozens to several hundred employees for a typical internal assistant, depending on how often each employee uses it and how long their sessions tend to be, so cost per user for lightly used tools can end up quite low once shared efficiently. Heavy users who run long conversations, upload large documents, or use the assistant continuously consume disproportionately more capacity than occasional users, so a per-user average can mask significant variance. Quantization and batching further improve how many users a given GPU footprint can support, lowering cost per user without adding hardware. Actual cost per user should be measured from real usage logs after a pilot period rather than estimated purely in advance. Nanobase AI sizes self-hosted assistant infrastructure against a client's actual expected user count and usage intensity to produce a realistic per-user cost figure.

A shared-cost curve, not a per-unit price

The instinct to think of cost per user like a per-seat software license misses the actual shape of the economics. A self-hosted assistant's infrastructure cost is largely fixed up to a concurrency ceiling, so cost per user is total monthly infrastructure cost divided by active users, and that number falls steeply as more users share the same GPU footprint, right up until the ceiling forces an additional GPU. This is a shared-cost curve, closer to a utility than a subscription.

A worked scaling table

Using an illustrative fixed monthly infrastructure cost for a single-GPU deployment sized for moderate concurrency (as of 2026, verify current GPU rental or amortized purchase pricing):

Cost per user does not fall smoothly forever; it drops within a capacity step and then jumps back up when the next GPU is added, which is why sizing to the right side of a capacity step, not the edge of it, produces the best economics.

What a Pareto skew does to the average

A single blended cost-per-user figure hides enormous variance in practice, since usage of an internal assistant typically follows a Pareto-like pattern where a minority of users generate the majority of tokens. A heavy user running long conversations, uploading large documents, or querying continuously throughout the day can consume ten times or more the GPU capacity of a light user checking in occasionally. The blended average is useful for budgeting the whole deployment, but it should never be used to estimate the marginal cost of adding one more specific user, since that marginal cost depends entirely on how heavily that particular user is expected to use the assistant. Measuring usage distribution from real logs after a pilot period, not assuming a uniform pattern, is what makes a per-user cost figure actually useful for planning.

Levers that lower cost per user without adding GPUs

1. Quantize the model to FP8 or INT4 to free VRAM, allowing more concurrent sessions on the same hardware without a quality cliff for most business tasks.
2. Increase batch size in the serving engine (vLLM, TensorRT-LLM) so more requests share compute simultaneously rather than queuing sequentially.
3. Cap or gently throttle unusually long conversations or document uploads from the heaviest users, since they disproportionately consume shared capacity.
4. Use prompt caching for any shared system prompt or reused context, lowering the effective compute per request across all users.
5. Route routine, well-defined queries to a smaller co-hosted model, reserving the primary model's capacity for tasks that need it, as covered in small language models vs frontier cost.

Frequently asked questions

How many employees can one GPU realistically support?

It varies widely with session length and frequency, but a single well-configured GPU running a 70B-class model at FP8 can often support employee counts ranging from dozens to a few hundred for a typical internal assistant; the only reliable way to know the number for a specific workforce is measuring real pilot usage.

Does cost per user include the engineering and operations time to run the assistant?

A complete figure should, since patching, monitoring, and tuning are real recurring costs, not just GPU-hours. Excluding operations time understates cost per user and can make a self-hosted assistant look cheaper than it actually is on a fully loaded basis.

Is cost per user a good metric for comparing self-hosted against a per-seat product?

It is a useful starting point but should be compared against the same population's usage pattern under a per-seat license, since the two curves behave very differently, as covered in ChatGPT Enterprise vs private LLM cost at scale, before drawing a final conclusion.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, sizes self-hosted assistant infrastructure against a client's actual expected usage distribution, not a flat average, then applies quantization, batching, and caching to keep cost per user low as adoption grows. See solutions for the full internal assistant deployment stack.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.