How do we serve multiple LoRA adapters on one GPU?

Serving many LoRA adapters on a single GPU is done by keeping one copy of the base model resident in memory and dynamically loading the small adapter matrices per request, which inference engines like vLLM support natively through multi-LoRA serving, so a single deployment can route different requests to different adapters, such as one per customer or one per task, without duplicating the full model weights for each. Because each adapter is typically only tens to a few hundred megabytes, dozens of adapters can be held in GPU memory simultaneously alongside one base model, compared to the impossibility of loading dozens of full fine-tuned copies of the same model. Batching requests that use different adapters together is more complex than batching identical requests, so throughput per adapter is somewhat lower than serving a single fine-tuned model, and very high adapter counts eventually hit scheduling overhead. This approach is the standard pattern for multi-tenant deployments where each customer or business unit needs slightly different behavior from the same base model. Nanobase AI configures multi-LoRA serving on vLLM or NVIDIA NIM so clients can run many customized behaviors on shared GPU capacity instead of one GPU per customization.

The mechanics behind multi-adapter serving

Engines that support multi-LoRA serving, such as vLLM, use batched adapter kernels that let a single forward pass apply different LoRA matrices to different sequences within the same batch, rather than requiring the whole batch to share one adapter. Each incoming request carries an adapter identifier, the engine looks up the matching low-rank matrices already resident in GPU memory, and the base model's forward computation is shared across every request in the batch regardless of which adapter it carries. This is fundamentally different from naive approaches that swap the whole model or reload weights per request, which would make per-adapter latency unworkable at any real request volume.

The base model computation is shared while only the small adapter delta varies per request, which is what makes dozens of adapters practical on a single GPU. This also means throughput scales close to single-model throughput as long as the adapters stay small relative to the base model's own compute cost.

Sizing the adapter memory budget

Adapter size follows directly from the LoRA formula: parameters per targeted module equal rank × (input dimension + output dimension), multiplied by the number of layers and target modules. For a 7B-class model with a 4,096 hidden size, four targeted attention projections and 32 layers, the numbers scale predictably with rank.

Against a base model occupying tens of gigabytes, even fifty rank-32 adapters add only a few gigabytes on top, which is why the adapter memory budget is rarely the constraint; the GPU's headroom after loading the base model weights and reserving KV cache is what actually limits adapter count.

Running it in production

Treating adapter registration and rollout as a first-class operational process, not an afterthought, is what keeps a multi-adapter deployment maintainable as tenant count grows.

1. Pick a serving engine with native multi-LoRA support (vLLM exposes --enable-lora, --max-loras for concurrently active adapters, and --max-lora-rank for the ceiling on any single adapter).
2. Register each adapter with a stable identifier that maps to a tenant, task or customer, and route requests to that identifier at the API layer rather than hardcoding a model name.
3. Load new or updated adapters at runtime through the engine's adapter API instead of restarting the deployment, since adapters are small enough to load in seconds.
4. Monitor per-adapter latency and error rates separately, because a single misbehaving adapter (for example, one trained at a much higher rank than the rest) can disproportionately affect batch scheduling.
5. Set a hard cap on simultaneously active distinct adapters per batch, since kernel efficiency degrades once the number of distinct adapters in a single batch grows very large.

Where this approach hits its limits

Multi-LoRA serving assumes every adapter shares the same base model and quantization scheme; adapters trained against different base model versions cannot be mixed in one deployment. Very high ranks (128 and above) reduce the efficiency advantage over simply running separate models, since the adapter computation starts to rival the savings from sharing base weights. Quantized base models (via QLoRA-style 4-bit weights) also add a dequantization step that most engines handle for LoRA inference, but it is worth confirming your chosen framework supports that exact combination before committing to a design. For teams choosing between this pattern and a single merged model per tenant, the LoRA merge trade-offs article covers the opposite end of that decision.

Frequently asked questions

How many LoRA adapters can one GPU realistically serve?

It depends on adapter rank and base model size, but tens of adapters at rank 16-32 on a single high-memory GPU is common in production, since each adapter typically adds only tens of megabytes. The practical ceiling is usually set by KV cache headroom and per-batch kernel overhead rather than raw adapter storage.

Does adding more adapters slow down every request?

Adding resident but inactive adapters has negligible cost. Performance only degrades when many distinct adapters are active within the same processing batch simultaneously, since the batched kernel has to handle more distinct low-rank computations per step, an effect worth measuring directly under your own traffic mix rather than assuming.

Can adapters be updated without downtime?

Yes. Because adapters are small independent files, most multi-LoRA engines support loading a new or retrained adapter version at runtime and redirecting new requests to it, without restarting the base model deployment or affecting in-flight requests on other adapters currently in production.

How Nanobase AI helps

Nanobase AI designs and deploys multi-tenant LoRA serving on self-hosted GPU infrastructure, from choosing the right rank and target modules per adapter to configuring vLLM or TensorRT-LLM for production-grade adapter routing and monitoring. That includes sizing the GPU footprint so adapter count, KV cache and base model weights fit together rather than being sized separately after the fact.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.