Adding rate limits and cost controls to LLM tool use starts at the gateway or proxy layer sitting between the model and the tools it calls, since that is the single point where every call can be measured and throttled before it happens. Practical controls include per-user and per-tool call quotas, a token or dollar budget attached to each session or conversation that stops further calls once exhausted, and circuit breakers that halt an agent after a set number of consecutive tool calls to catch runaway loops, a real risk with autonomous multi-step agents that can otherwise call the same tool dozens of times chasing an unreachable goal. Cost attribution tags on each call, by team, project or use case, make it possible to see where spend is actually going rather than facing a single opaque monthly bill. Alerting thresholds that notify an engineering team well before a hard limit is hit give time to react to unusual usage patterns rather than discovering them after the fact. Nanobase AI builds these controls into the same gateway layer it uses for logging, so cost and audit data come from one consistent source.
Tool-use costs behave differently from token costs
Token-based cost is predictable within a range: a longer prompt or response costs proportionally more, and it is bounded by context window size. Tool-use cost is not bounded the same way. An autonomous agent chasing an unreachable goal can call the same tool dozens of times in a loop, each call incurring its own latency, its own token cost for the surrounding reasoning, and potentially its own side effect on a downstream system, such as creating duplicate tickets or repeatedly querying an expensive report. A single misbehaving agent session can generate more cost and more system load in an hour than a hundred well-behaved conversations combined, which is why tool-use controls need their own layer separate from ordinary token-based rate limiting.
Layered limit design
No single limit catches every failure mode, so a workable setup applies different limits at different scopes:
| Layer | Example limit | What it stops |
|---|---|---|
| Per-user | N tool calls per hour | One user's runaway session or misuse |
| Per-tool | N calls per minute to a specific expensive tool | A single slow or costly backend from being hammered |
| Per-session | Dollar or token budget per conversation | A single agent run from spiraling past its intended scope |
| Per-team/project | Monthly spend cap with alerting | Budget surprises at the organizational level |
No single layer catches every failure mode on its own; it's the combination that closes the gaps any one limit would miss.
Circuit breakers for runaway loops
Quotas cap total volume, but a circuit breaker catches the specific failure pattern of an agent calling the same tool repeatedly without making progress. A simple, effective version halts execution after a fixed number of consecutive tool calls within one agent turn, or after a fixed number of calls to the same tool with near-identical arguments, and returns control to a human rather than letting the loop continue indefinitely. This single guardrail addresses a real and common risk with autonomous multi-step agents, since without it, a poorly scoped goal can turn into hundreds of calls before anyone notices. A circuit breaker is the one control that catches a failure pattern quotas alone cannot see.
An example policy shape
A gateway policy does not need to be complex to be effective. A minimal configuration might express the layered limits above as plain rules attached to a user or team identity:
policy:
user_default:
max_tool_calls_per_hour: 200
max_session_spend_usd: 5
tool_overrides:
generate_quarterly_report:
max_calls_per_minute: 2
circuit_breaker:
max_consecutive_calls: 15
max_identical_calls: 5The specific numbers matter less than having every limit defined in one reviewable place, so a platform team can tune them based on observed usage rather than guessing at rollout.
Frequently asked questions
Where should rate limits actually live?
At the gateway or proxy layer sitting between the model and the tools it calls, since that is the single point where every call passes through and can be measured before it executes. Implementing limits separately inside each individual tool or target system leads to inconsistent enforcement and makes cost attribution much harder to reconstruct.
How do we know what limits to set initially?
Start with observed usage from a pilot period rather than guessing, set limits somewhat above that baseline, and tighten them once you have data on normal versus anomalous patterns. Pair every limit with alerting so a threshold breach notifies a team before it becomes a hard block that interrupts real work.
Do circuit breakers apply to single tool calls or whole agent sessions?
They apply best at the session or agent-run level, tracking a pattern across multiple calls rather than judging any single call in isolation. A circuit breaker that only looks at one call at a time cannot detect a loop, since each individual call in a loop may look perfectly legitimate on its own.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, builds rate limiting and cost controls into the same gateway layer it uses for tool-call logging and auditing, so cost, usage, and audit data all come from one consistent source rather than three disconnected systems. This is part of the broader AI gateway work we do for clients running multiple teams and tools against shared model infrastructure. See a working example in a live demo.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.