Deploying AI agents on Kubernetes generally means packaging the agent's orchestration logic, which decides tool calls and manages state, as a stateless or checkpointed service that scales independently from the model inference layer, since the two have very different resource profiles. The orchestration service itself, whether built on LangGraph, the Claude Agent SDK or a custom loop, typically runs as a standard containerized deployment behind a queue or API gateway, scaled horizontally with the Horizontal Pod Autoscaler based on request volume rather than GPU usage. If you self-host the underlying models, that inference layer runs separately using vLLM or NVIDIA NIM containers scheduled through the NVIDIA GPU Operator, with its own scaling policy tied to token throughput and queue depth rather than the agent orchestration pods. Long-running or multi-step agent tasks benefit from persisting state in an external store, such as Redis or a database, rather than in pod memory, so a pod restart or rescheduling event does not lose an in-progress task. Network policies should restrict what each agent pod can reach, particularly for agents with code execution or web access, treating the cluster's own network segmentation as another safety layer. Nanobase AI designs and operates these Kubernetes-based agent and GPU inference deployments for clients running fully on-premise or hybrid infrastructure.
Splitting the deployment into two distinct workloads
Treat an agent deployment as two workloads with opposite scaling behavior, not one. The orchestration layer, which decides tool calls, holds conversation and task state, and calls out to a model, is CPU-bound and stateless-friendly, so it scales on request volume like any web service. The inference layer, if models are self-hosted rather than called through an API, is GPU-bound and scales on token throughput and queue depth, an entirely different signal. Deploying both as one tightly coupled service almost always wastes GPU spend, since the orchestration pods rarely need the GPU nodes they end up scheduled onto. Keeping them as separate deployments with separate node pools and separate autoscaling policies is the pattern that avoids this.
A reference resource layout
| Component | Runs as | Scales on | Typical node type |
|---|---|---|---|
| Agent orchestration service | Stateless Deployment behind an API gateway or queue | Request volume, HPA on CPU/RPS | Standard CPU nodes |
| Model inference (vLLM, NVIDIA NIM) | GPU-scheduled Deployment via the NVIDIA GPU Operator | Token throughput, queue depth | GPU nodes (H100, H200, RTX PRO) |
| Task state store | Managed Redis or Postgres, not pod memory | N/A, sized for throughput | Separate managed or stateful service |
| Sandbox execution pods | Short-lived Jobs or Pods per task, network-restricted | Concurrent task count | Isolated node pool with tight NetworkPolicy |
A minimal resource request and limit block for the orchestration deployment keeps a single runaway task from starving the node:
resources:
requests: { cpu: "250m", memory: "512Mi" }
limits: { cpu: "1", memory: "1Gi" }Why external state matters more than it seems
An agent pod that keeps a multi-step task's state only in memory loses that task entirely the moment Kubernetes reschedules the pod, whether from a node drain, a deployment rollout, or a crash. Writing task state to Redis or a database after each meaningful step, not only at the end, is what makes an agent deployment survive normal cluster operations rather than treating every rollout as a risk to in-flight work. This also enables horizontal scaling cleanly, since any orchestration pod can pick up any task by reading its state rather than requiring sticky routing back to the pod that started it.
Network and namespace isolation for agent workloads
Agents that execute generated code or browse the web need tighter network policy than a typical microservice. A dedicated namespace for sandboxed execution, with a default-deny NetworkPolicy and narrow, explicit egress allowlists per task type, contains a compromised or misbehaving agent to that namespace rather than the wider cluster. Separate service accounts per agent type, scoped through Kubernetes RBAC to only the secrets and APIs that agent's tools require, extend the same least-privilege principle that should already govern the agent's application-level tool permissions. A dedicated namespace with default-deny egress contains a compromised agent task to that namespace instead of the wider cluster.
Choosing between the GPU Operator and Slurm underneath
If GPU inference runs alongside the agent layer on the same cluster, the scheduler choice affects both. The NVIDIA GPU Operator fits teams that want inference treated as just another Kubernetes workload, sharing the same deployment and observability tooling as the agent services, while Slurm remains common where GPU training or batch jobs already dominate the cluster and inference is added alongside that existing scheduler. See the deeper comparison in Kubernetes GPU Operator vs Slurm for the tradeoffs on job types this decision actually turns on. The right scheduler follows whichever workload, inference or existing batch and training jobs, already dominates the cluster, not a blanket preference.
Frequently asked questions
Should agent orchestration pods ever share a node with GPU inference pods?
Generally no. Orchestration pods are cheap CPU workloads and scheduling them onto expensive GPU nodes wastes that capacity; use node taints and tolerations so GPU nodes are reserved for inference and sandbox workloads that actually need them.
How do we autoscale the agent orchestration layer correctly?
Scale the Horizontal Pod Autoscaler on request rate or queue depth rather than CPU utilization alone, since an agent task's CPU use is often low even while it waits on a slow tool call or model response, which would otherwise mask real load.
Do long-running agent tasks need a different deployment pattern than short ones?
Yes. Tasks lasting minutes to hours should run as background workers reading from a queue with checkpointed state, not as long-held HTTP requests, since no ingress or load balancer should keep a connection open for hours.
What is the biggest security gap teams miss when deploying agents on Kubernetes?
Unrestricted egress from sandbox or tool-execution pods. Default-deny NetworkPolicies with explicit allowlists close the most common path for a compromised agent task to exfiltrate data, and this is frequently skipped because it works fine in development where no policy is enforced.
How Nanobase AI helps
Nanobase AI designs and operates Kubernetes-based agent deployments end to end, from orchestration services and external state stores through GPU-scheduled inference using the NVIDIA GPU Operator, for clients running fully on-premise or hybrid infrastructure. As an NVIDIA Inception Program member, the team sizes and tunes both layers so agent orchestration and model inference scale on the signals that actually drive their cost and latency, rather than a single generic autoscaling rule.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.