How do I enable tool calling in vLLM or SGLang?

In vLLM, tool calling is enabled by starting the server with enable-auto-tool-choice and specifying a tool-call-parser that matches the model's chat template, such as llama3_json, hermes, mistral, or qwen, since different model families were fine-tuned to emit function call blocks in slightly different formats and vLLM needs the right parser to convert that raw text into a structured tool_calls field in the API response. You then pass your tool or function definitions in the tools parameter of the chat completions request exactly as with the OpenAI API, and the model decides whether to respond with text or a tool call based on its training. SGLang supports the equivalent capability through its own function-calling implementation, again requiring a model actually trained for tool use, since prompting alone rarely produces reliable structured tool calls from a model that was not fine-tuned for it. Model choice matters more than engine choice here: Llama 3.1 and later, Qwen 2.5 and later, and Hermes fine-tunes all have solid native tool-calling support, while older or smaller models often need heavier prompt engineering to get consistent results. Testing with your actual tool schemas before production is essential, since reliability varies noticeably across model families. Nanobase AI, a Silicon Valley AI engineering company, configures and validates tool-calling pipelines as part of its AI agent deployments.

Why the parser flag is the part that actually breaks

Enabling tool calling in vLLM looks deceptively simple: one flag turns the feature on. What actually determines whether it works reliably is a second, model-specific flag, the tool-call parser, and getting this wrong is the most common cause of "tool calling doesn't work" reports that are actually a parser mismatch rather than a model capability gap.

The model was fine-tuned to emit function-call blocks in one specific text format, and the parser's job is converting that raw text into the structured tool_calls field the API response promises; the wrong parser means the raw text passes through unparsed or gets parsed incorrectly.

Model family to parser mapping

Always check your specific vLLM version's supported parser list before deployment, since new parsers are added as new model families ship with native tool-calling fine-tunes, and a model released after your vLLM version may not yet have a matching parser available.

Setup walkthrough

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json

1. Identify your model's family and confirm it was actually fine-tuned for tool use; model choice matters more than engine choice here.
2. Check the current parser list for your vLLM or SGLang version and select the one matching your model family.
3. Start the server with tool choice enabled and the matched parser flag set.
4. Pass your tool or function definitions in the tools parameter of the chat completions request, in the same shape as the OpenAI API.
5. Test with your actual production tool schemas, not a toy example, since reliability varies noticeably across model families and schema complexity.
6. Verify the response's tool_calls field is populated correctly rather than the tool call appearing as unparsed text in the message content, which is the direct symptom of a parser mismatch.

SGLang's equivalent, and why model choice still dominates

SGLang implements the same capability through its own function-calling support, requiring a model actually trained for tool use in the same way vLLM does; prompting an untrained model to emit function-call-shaped text works inconsistently regardless of which engine parses the result. This is why model selection, not engine selection, is the first decision to get right: Llama 3.1 and later, Qwen 2.5 and later, and Hermes fine-tunes all have solid native tool-calling support, while older or smaller general-purpose models often need heavier prompt engineering to get consistent results, and even then remain less reliable than a natively tool-tuned model.

What reliability testing should actually check

Reliable tool calling is often paired with structured output and JSON mode in agentic pipelines, since both aim at getting programmatically consumable output from the model rather than free text.

Frequently asked questions

Can we use tool calling with a model that wasn't specifically fine-tuned for it?

Prompting alone can sometimes produce usable results for simple cases, but reliability drops noticeably compared with a natively tool-tuned model, especially as the number of available tools or the complexity of arguments grows; test thoroughly before depending on this for production.

What happens if the tool-call parser doesn't match the model?

The model's function-call output typically appears as unparsed text in the response content rather than in the structured tool_calls field, which breaks any downstream code expecting the OpenAI API's structured format, often silently rather than with an obvious error.

Does tool calling work with streaming responses?

Support for combining tool calling with streaming has improved across recent vLLM and SGLang releases, but behavior can vary by version and parser, so test this combination specifically if your application both streams and relies on tool calls.

How do we know if a new model release has vLLM tool-calling support yet?

Check the current release's documented parser list and any recent release notes; support for a newly released model family's specific tool-call format sometimes lags the model's own release by days to weeks while a matching parser is added.

How Nanobase AI helps

Nanobase AI configures and validates tool-calling pipelines against a customer's actual tool schemas and model choice as part of its AI agent deployment work, since reliability depends on matching model, parser, and schema together rather than any single configuration flag.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.