Sizing GPUs for 1,000 employees using an internal AI assistant starts with estimating realistic peak concurrency rather than total headcount, since usage logs from comparable deployments typically show only 5 to 15 percent of users active in any given moment even during business hours, putting peak concurrency in the range of 50 to 150 simultaneous sessions for this size of organization. From there, the model choice drives GPU count: a 7B to 14B assistant in FP8 can often cover that concurrency on one to two H100 or H200 GPUs, while a 70B model for higher-quality responses typically needs four or more GPUs with tensor parallelism to hold both the weights and enough KV cache headroom for peak load. Average query length, whether the assistant retrieves documents through RAG, and whether responses are long-form all shift the estimate meaningfully. Because assumptions about concurrency are the biggest source of error, running a pilot with a smaller group and measuring actual usage before committing to full-scale hardware is the safer path. Nanobase AI, an NVIDIA Inception Program member, builds this kind of sizing model from real usage data before recommending a GPU count for enterprise-wide rollout.
A phased hardware plan, not one big purchase
Sizing for 1,000 employees in one step invites two failure modes: over-buying against an assumed peak that never materializes, or under-buying because the pilot's usage pattern does not represent the full rollout. A phased plan avoids both by scaling hardware alongside measured adoption.
| Phase | Scope | Estimated peak concurrency | Suggested GPU configuration |
|---|---|---|---|
| Pilot | 50-100 employees, one department | 5-15 sessions | 1x H100 or RTX PRO 6000, 7B-14B model |
| Early rollout | 300-500 employees | 20-60 sessions | 1-2x H100/H200, 7B-32B model depending on findings |
| Full rollout | 1,000 employees | 50-150 sessions | 2-4x H100/H200 with tensor parallelism for a 70B model, or fewer for a smaller model |
| Growth buffer | Beyond initial 1,000 | Plan for 20-30% above measured peak | Add GPUs incrementally as usage data justifies |
Key takeaway: treating the rollout as three sequential phases, each informed by the last, produces a far more defensible hardware plan than sizing for 1,000 users from day one.
Why 5 to 15 percent active concurrency is the right planning assumption
Usage data from comparable internal assistant deployments consistently shows that only a fraction of licensed or eligible users are actively generating a request at any given moment, even during peak business hours, because employees read responses, switch to other tasks, or are between messages far more often than they are actively waiting on the model. For 1,000 employees, this puts realistic peak concurrency in the range of 50 to 150 simultaneous sessions rather than anything close to 1,000, and sizing hardware against the full headcount would badly overprovision the deployment. The pilot phase's most valuable output is not proving the assistant works but measuring this organization's actual concurrency ratio, since it varies by role mix, use case and how deeply the tool gets embedded into daily workflows.
Key takeaway: the pilot phase exists to measure your organization's real concurrency ratio, which is the single number that drives every later hardware decision.
What to measure during the pilot before scaling
- Peak concurrent sessions during the busiest observed hour, not the daily average.
- Average and 95th-percentile prompt and response length, since both directly affect KV cache and throughput.
- The fraction of queries that require document retrieval (RAG) versus general knowledge, since retrieval adds latency and a separate embedding workload.
- Time-to-first-token and total response time as experienced by users, correlated with concurrent session count at that moment.
- Which departments or use cases drive the most usage, since this often reveals that a smaller, cheaper model handles most traffic well and only specific use cases need the larger model.
Key takeaway: the metrics gathered during a small pilot, not assumptions about headcount, are what should determine the full-rollout GPU count.
Why headcount-based sizing tends to fail in both directions
Sizing directly from headcount, for example assuming 1,000 employees means provisioning for 1,000 concurrent sessions, produces hardware that sits mostly idle at a large capital cost, since actual concurrency almost never approaches total headcount. The opposite mistake, assuming a small internal tool will stay small, is just as common: a pilot that works well tends to spread by word of mouth faster than IT budgets anticipate, and a deployment sized only for the pilot's own department can degrade sharply once other teams start using it without anyone updating the hardware plan. Both failure modes trace back to the same root cause: sizing from an assumption rather than from measured, phase-appropriate data.
Key takeaway: both over-provisioning from headcount and under-provisioning from a too-small pilot come from skipping the measurement step, not from a wrong formula. For the smaller-scale version of this same calculation, see sizing GPUs for 100 concurrent chatbot users.
Frequently asked questions
Should the full rollout use the same model as the pilot?
Not necessarily. A pilot often reveals that a smaller, cheaper model satisfies most queries well, with only certain departments or use cases genuinely needing a larger model's quality, which can lead to a mixed deployment rather than one model sized for the hardest case across everyone.
How long should a pilot run before scaling?
Long enough to capture a full weekly usage cycle and at least one unusually busy period, commonly four to eight weeks, since usage patterns during the first few days rarely reflect steady-state adoption once employees build the tool into their routine.
Does RAG change the GPU sizing for this scenario?
Yes, a RAG-based assistant adds embedding model inference and vector search latency on top of the LLM's own compute and memory needs, so sizing has to account for the embedding workload separately rather than folding it into the LLM's GPU budget.
What is the risk of skipping the pilot phase entirely?
Committing to full-scale hardware based on headcount alone risks either significant over-spending on idle capacity or an under-provisioned launch that degrades user trust in the tool before adoption even takes hold, which is harder to recover from than a slower, staged rollout.
How Nanobase AI helps
Nanobase AI, an accepted member of the NVIDIA Inception Program, builds this phased sizing model from real usage data gathered during a pilot, rather than recommending a GPU count for enterprise-wide rollout upfront. We help identify which use cases need a larger model and which are well served by a smaller one, and design the growth path from pilot to full deployment. Our best open-weight models for enterprise guide is a useful starting point for the model selection question.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.