How do I serve a vision-language model like Qwen 3 VL with vLLM?

vLLM serves vision-language models like Qwen 3 VL by loading the model with its multimodal processor enabled and accepting images alongside text in the same chat completions request, passing each image as a base64-encoded data URL or a hosted image URL inside the content array, matching OpenAI's vision API format. You typically need to set limit-mm-per-prompt to cap how many images a single request can include, since each image consumes meaningfully more GPU memory and compute than an equivalent amount of text through the vision encoder and image token expansion, and this setting protects against memory exhaustion from requests with many high-resolution images. Vision-language models generally need more GPU memory headroom than a text-only model of similar parameter count, both for the vision encoder weights and for the larger effective sequence length images translate into once encoded as tokens, so capacity planning should account for realistic image sizes and counts per request rather than text-only benchmarks. Video input, where supported, multiplies this further since frames are sampled and encoded similarly to multiple images. Batching behavior with mixed image and text-only requests also needs testing, since it can affect throughput more than pure text workloads. Nanobase AI deploys multimodal serving stacks for document understanding and visual inspection use cases built on models like Qwen VL.

What actually happens to an image between request and response

An image sent to a vision-language model does not stay an image for long. vLLM's multimodal processor runs it through the model's vision encoder, which turns it into a block of embedding vectors that get spliced into the token sequence the language model actually attends over. A single moderate-resolution image commonly expands into hundreds of these "image tokens," which is why a request with one image can consume as much context budget and compute as a fairly long block of text, even though the user only sent one file.

This is the detail most teams miss when they size a multimodal deployment using text-only benchmarks: two requests with the same word count in their text portion can have wildly different compute cost if one carries three high-resolution images and the other carries none.

Image tokens, not image file size, are what actually consumes your context window and compute budget.

The request format and the flags that keep it safe

vLLM accepts images through the same chat completions endpoint used for text, matching OpenAI's vision API shape: each message's content becomes an array with a text object and one or more image_url objects, where the URL is either a hosted image link or a base64 data URL.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-vl",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this defect"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}}
      ]
    }]
  }'

Two flags matter most in production. --limit-mm-per-prompt caps how many images (and, where supported, video frames) a single request can attach, which is the main defense against a request with dozens of high-resolution images exhausting GPU memory on its own. --max-model-len needs to account for realistic image-token expansion, not just the expected text length, or requests will silently truncate context that the user assumed was preserved.

Treat limit-mm-per-prompt as a memory safety control, not an API nicety, and set it before your first production request.

Sizing memory for vision workloads

A vision-language model needs meaningfully more headroom than a same-parameter-count text model, both because the vision encoder itself occupies memory and because a handful of images can push effective sequence length well past what the same conversation would need in text alone. Capacity planning should use your application's realistic image count and resolution per request, not a text-token estimate with images bolted on afterward.

Size vision deployments on realistic image volume per request, since that number, not parameter count, drives the memory difference from text-only serving.

Video input and other engines

Where a model and engine support video, frames are typically sampled at some rate and each sampled frame is encoded much like a separate image, which multiplies the image-token cost by the frame count. This makes video by far the most expensive input type per request and the one most worth capping explicitly through configuration rather than trusting default limits.

vLLM is not the only option for vision-language serving. SGLang supports many of the same multimodal architectures with its own scheduler optimizations, and NVIDIA NIM packages multimodal models with pre-tuned defaults for teams that want less configuration surface. TensorRT-LLM also supports select vision-language architectures for teams that have already standardized on it for text models and want one engine across both. The right choice usually follows whichever engine already serves your text models, rather than treating multimodal as a reason to run a second stack.

Standardizing on one engine for both text and vision workloads is usually simpler than running a separate stack just for multimodal.

Frequently asked questions

How many images can one request safely include?

There is no universal number; it depends on GPU memory, image resolution, and your concurrency target. Start with --limit-mm-per-prompt set to a small number like 4–8, load test with realistic images, and raise it only if memory headroom allows without degrading concurrent throughput.

Does batching still work well with mixed image and text-only requests?

Continuous batching still functions, but throughput characteristics change because image-heavy requests consume disproportionately more compute and memory per slot in the batch. Load test with a realistic mix of image and text-only traffic rather than assuming pure text-only benchmarks transfer.

Can I run a vision-language model on the same GPU as a text-only model?

Yes, with MIG partitioning or multi-model serving, provided you size each partition for its own workload's memory profile. The vision model's partition needs the extra headroom described above; treating both models identically will undersize the multimodal one.

Do quantized vision-language models lose accuracy on the vision side?

Quantization (AWQ, GPTQ, FP8) typically targets the language model weights; vision encoder behavior is less consistently affected across implementations, so validate image-understanding quality on your own test images after quantizing rather than assuming text-quality benchmarks generalize.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys multimodal serving stacks for document understanding, visual inspection, and defect-detection use cases built on models like Qwen 3 VL, sizing memory and concurrency around real image and video volume rather than text-only estimates. This work sits alongside our broader on-premise LLM deployment practice; see solutions for the full stack, or book a demo to see multimodal serving in action.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.