Running Ollama on a server for a team starts with installing it on a machine with a capable GPU, setting the OLLAMA_HOST environment variable to listen on the network rather than only localhost, and opening the appropriate port behind your firewall or VPN rather than exposing it directly to the internet, since Ollama has no built-in authentication. Put a reverse proxy such as nginx or Caddy in front of it to add API key or basic authentication, TLS termination, and request logging, since these are things Ollama itself does not provide out of the box. Set OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS to control how many requests and models run concurrently, keeping in mind that Ollama's parallel request handling, while improved in recent versions, still does not match the continuous batching depth of vLLM or SGLang under heavy concurrent load. This setup works well for a team of a few people to a few dozen with moderate, bursty usage; once usage grows into hundreds of concurrent requests or strict latency SLAs, migrating to vLLM or NVIDIA NIM behind the same OpenAI-compatible interface is the natural next step. Nanobase AI, a Silicon Valley enterprise AI engineering company, sets up these shared internal model servers and the migration path beyond them as usage scales.
Why the default Ollama install is not team-ready
A fresh Ollama install binds to localhost only and has no authentication layer at all, which is fine for one developer on one laptop and a real problem the moment a second person needs access. The gap is not a bug, Ollama was built first as a single-user local tool, so making it team-usable is a deliberate setup step rather than something the defaults handle for you.
Treat Ollama's out-of-the-box configuration as a single-user starting point, not a server-ready default, and budget setup time for the gaps below accordingly.
Setup steps for a shared team server
- Install Ollama on a machine with a capable GPU. Sizing follows the same rules as any other serving engine, matched to your largest expected model and concurrency.
- Set
OLLAMA_HOST=0.0.0.0(or a specific internal interface) so the service listens beyond localhost, and confirm the port is only reachable inside your VPN or private network, never exposed directly to the public internet. - Put a reverse proxy in front of it. Nginx or Caddy handles TLS termination, API key or basic authentication, and request logging, none of which Ollama provides natively.
- Set
OLLAMA_NUM_PARALLELto control how many requests one loaded model processes concurrently, based on GPU memory headroom and your expected simultaneous user count. - Set
OLLAMA_MAX_LOADED_MODELSto cap how many different models can be resident in GPU memory at once, preventing an unexpected model pull from evicting a model your team depends on. - Run it as a managed service (systemd unit or container with restart policy) rather than a foreground process, so it survives reboots and crashes without manual intervention.
- Add basic monitoring on GPU utilization, memory, and request latency before opening access broadly, so degradation is visible before users start complaining.
Steps 2 through 4 are the ones that actually turn a single-user tool into a usable shared service; skipping any of them tends to surface as either a security incident or a mystery outage later.
Key environment variables at a glance
| Variable | Purpose | Typical consideration |
|---|---|---|
OLLAMA_HOST | Network interface and port to bind | Internal network only, never public |
OLLAMA_NUM_PARALLEL | Concurrent requests per loaded model | Set based on GPU memory and expected simultaneous users |
OLLAMA_MAX_LOADED_MODELS | Number of models kept resident at once | Cap to prevent unexpected eviction of an in-use model |
OLLAMA_KEEP_ALIVE | How long an idle model stays loaded | Balance memory reuse against reload latency for bursty usage |
OLLAMA_MODELS | Model storage directory | Point at fast local storage with enough disk for your model set |
These five variables cover most of what a team deployment needs beyond the reverse proxy, and getting them wrong tends to show up as either wasted GPU memory or unnecessary reload latency rather than an outright failure.
Where this setup stops scaling
This configuration comfortably serves a team from a few people to a few dozen with moderate, bursty usage, chat-style queries, occasional batch jobs, nothing approaching sustained heavy concurrency. The limiting factor is the same one that separates Ollama from purpose-built serving engines everywhere else: its parallel request handling, while improved substantially in recent versions, still does not match the continuous batching depth of vLLM or SGLang under sustained high concurrent load.
The practical signal to watch for is request queuing that grows under normal working-hours traffic rather than only during genuine spikes. When that happens consistently, migrating to vLLM or NVIDIA NIM behind the same OpenAI-compatible API surface is the natural next step, since client code calling an OpenAI-style endpoint generally does not need to change.
Plan the migration path to vLLM or NIM before you need it, since the OpenAI-compatible interface makes that transition far less disruptive than switching client integrations from scratch would be.
Frequently asked questions
Does Ollama support API key authentication natively?
No, Ollama itself has no built-in authentication. API keys, basic auth, or OAuth need to be implemented at a reverse proxy or API gateway layer in front of it, which is also where you would add per-team rate limiting if needed.
How much disk space should I plan for a team's model library?
Enough for every model variant your team actually uses concurrently, since Ollama keeps pulled models on disk even when not loaded into memory; a handful of 70B-class models at multiple quantization levels can add up to several hundred gigabytes.
Can multiple teams share one Ollama server safely?
Yes, with a reverse proxy issuing separate API keys per team and OLLAMA_NUM_PARALLEL sized generously enough that one team's burst does not starve another; without per-team rate limiting, a heavy user can still monopolize the shared queue.
When is it worth skipping Ollama and going straight to vLLM for a new team deployment?
When you already know concurrency will be high and sustained, or when strict latency SLAs exist from day one, since setting up vLLM correctly from the start avoids a later migration under production load.
How Nanobase AI helps
Nanobase AI sets up shared internal Ollama servers for teams starting their self-hosted LLM journey, including the reverse proxy, authentication, and monitoring layers above, and plans the migration path to vLLM or NVIDIA NIM as usage grows past what Ollama's concurrency model comfortably serves. See our on-premise deployment guide for the broader infrastructure picture, or book a demo to see the migration path in practice.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.