How do I handle rate limiting and quotas for internal LLM APIs?

Rate limiting and quotas for internal LLM APIs are best enforced at a gateway layer in front of your inference servers, using tools like LiteLLM, Kong, or Envoy to issue per-team or per-application API keys, then applying token bucket or sliding-window rate limits and monthly or daily token budgets tied to each key rather than relying on the inference engine itself to manage this. This protects shared GPU capacity from a single noisy team or a runaway agent loop starving other consumers, and it gives finance and platform teams the usage visibility needed to attribute GPU cost back to the teams generating it. Limits should be set on both requests per minute and tokens per minute, since a small number of very long requests can saturate a GPU just as effectively as many short ones, and vLLM's own request queue depth is a useful signal for when global capacity, not just individual quotas, is the constraint. For genuinely business-critical internal applications, a tiered quota system with priority queuing ensures they are not starved by lower-priority batch or experimental workloads sharing the same cluster. Alerting on quota exhaustion and queue depth before users notice degraded latency closes the operational loop. Nanobase AI builds these gateway, quota, and observability layers as part of internal LLM platform engagements.

Why the inference engine is the wrong place to enforce this

vLLM and similar engines are built to maximize throughput across whatever requests arrive, they have no concept of "team A" or "monthly budget" and should not be extended to have one. Enforcing quotas inside the engine means rebuilding authentication, accounting, and policy logic that a dedicated gateway already solves, and it couples business logic to infrastructure that should stay focused on inference performance.

The cleaner architecture puts a gateway layer in front of one or more inference backends, handling authentication, rate limiting, and usage accounting there, while the engines behind it stay simple and interchangeable.

Quota and rate-limit logic belongs in a gateway layer in front of inference, not inside the serving engine, which should stay focused purely on throughput and latency.

Gateway options and what each one covers

LiteLLM's proxy is worth calling out specifically because it was built for exactly this problem: issuing virtual API keys per team or application, setting per-key spend or token budgets, and routing across multiple backend models or engines through one OpenAI-compatible endpoint.

Purpose-built LLM gateways save the work of bolting token-aware accounting onto a general-purpose API gateway that was never designed to count tokens in the first place.

Requests per minute is not enough on its own

A rate limit set only on requests per minute misses the actual resource being consumed: a handful of very long requests, long prompts with long generated outputs, can saturate a GPU as thoroughly as a much larger number of short ones. Limits need to cover both requests per minute and tokens per minute (input and output) to actually reflect GPU load, and the token-based limit is usually the one that matters more for protecting shared capacity.

vLLM's own request queue depth is a useful cross-check here: if queue depth grows even while individual teams stay within their token quotas, that signals the constraint is total cluster capacity, not any one team's usage, which calls for a capacity or scaling response rather than tighter quotas.

Token-per-minute limits, not request counts alone, are what actually protect shared GPU capacity from a small number of long requests.

Building in priority without starving anyone

For genuinely business-critical internal applications sharing infrastructure with lower-priority experimental or batch workloads, a flat first-come-first-served queue risks the important application getting stuck behind a batch job someone kicked off overnight. A tiered quota system with priority queuing, where critical applications get a reserved capacity floor and everything else competes for the remainder, addresses this without requiring separate infrastructure per priority tier.

1. Classify each internal application or team into a priority tier based on business criticality.
2. Reserve a capacity floor (via separate quota pools or scheduler priority) for the highest tier.
3. Let lower tiers share remaining capacity, queuing or degrading gracefully under contention.
4. Alert on quota exhaustion and queue depth growth before users notice latency, not after.

A tiered priority system with a reserved capacity floor for critical applications prevents the common failure mode of an important service getting starved by a lower-priority batch job on shared infrastructure.

Frequently asked questions

Should rate limits differ by model, not just by team?

Yes, since a large model consumes far more GPU capacity per token than a small one; a team's quota is more meaningfully expressed as a token or cost budget that accounts for which model they are calling rather than a flat request count across models.

How do we attribute GPU cost back to teams using a shared cluster?

A gateway that logs token usage per API key, tagged to a team or application, gives finance and platform teams the raw data needed for cost attribution; this is typically built on the same logging the rate-limiting layer already needs for enforcement.

Does LiteLLM replace the need for Kong or Envoy?

Not necessarily; some deployments run LiteLLM specifically for LLM-aware routing and budgeting behind a more general gateway like Kong or Envoy handling broader API traffic, TLS, and network-level concerns.

What should happen when a team exceeds its quota?

This depends on business priority: a hard block with a clear error message works for most internal quotas, while critical applications might instead degrade to a smaller model or lower priority queue rather than failing outright, which requires the gateway to support that fallback logic explicitly.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception program member, builds gateway, quota, and observability layers as part of internal LLM platform engagements, sizing token-based budgets and priority tiers around each team's actual usage rather than generic per-user limits. This work pairs closely with our guidance on replacing the OpenAI API with self-hosted models and the LiteLLM gateway question in this same series.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.