# Serving with vLLM

> Serve an installed patch with static, allowlisted adapters, and why dynamic LoRA endpoints stay off.

Source: https://retoque.xyz/docs/vllm
Markdown: https://retoque.xyz/docs/md/vllm

vLLM can serve a LoRA adapter next to its base model. This path is documented but untested for published patches, and the runtime matrix on each patch page lists it as untested.

## Support status

Published evaluations run in transformers with PEFT. vLLM uses different kernels, batching and sampling code, so outputs may differ from the evaluated results even at temperature 0. Re-run your own checks before relying on a patch behind vLLM. Retoque doesn't publish vLLM results.

## Use static, allowlisted adapters only

Load adapters when the server starts, from directories the CLI installed and verified. Requests pick an adapter by name with the `model` field. Nothing is loaded or unloaded while the server runs.

```sh
retoque verify <slug>@<version>

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 \
  --enable-lora \
  --max-lora-rank 16 \
  --lora-modules <slug>=/srv/retoque/patches/<slug>/<version>
```

```sh
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "<slug>",
    "temperature": 0,
    "max_tokens": 128,
    "messages": [
      {"role": "system", "content": "<the system prompt the patch was evaluated with>"},
      {"role": "user", "content": "Today is 2026-03-02 (Monday).\nPay Halcyon Realty $4,200.00 for the office rent, due next Friday."}
    ]
  }'
```

- Pin `--revision` to the full commit in the manifest. A branch name can move.
- Set `--max-lora-rank` to at least the `adapter.rank` in the manifest.
- Mount the install directory read-only in the container. The CLI already makes the files read-only on disk.
- To serve the base model without the patch, send the base model name in `model`. That is the vLLM equivalent of `retoque disable`.

## Never expose dynamic LoRA endpoints

> [!warning] **Leave runtime LoRA updating off.** vLLM can be started with runtime LoRA updating, which adds endpoints that load and unload adapters from a path. Anyone who can call those endpoints can make your server load arbitrary files as weights and change what every other caller receives.

- Leave `VLLM_ALLOW_RUNTIME_LORA_UPDATING` unset on any public or shared server.
- If you need hot swapping, keep the management endpoints on a private, authenticated network, and only load directories that passed `retoque verify`.
- Put the inference API itself behind your own authentication. vLLM has no per-patch license checks.

## Licensing when you serve a patch

Serving a patch to other people is covered only if the version's license terms allow it. Check the commercial scope on the patch page before you deploy. A free developer license does not automatically cover a hosted service. See [Licenses and entitlements](https://retoque.xyz/docs/licenses).