Serving with vLLM

vLLM can serve a LoRA adapter next to its base model. This path is documented but untested for published patches, and the runtime matrix on each patch page lists it as untested.

Support status

Published evaluations run in transformers with PEFT. vLLM uses different kernels, batching and sampling code, so outputs may differ from the evaluated results even at temperature 0. Re-run your own checks before relying on a patch behind vLLM. Retoque doesn't publish vLLM results.

Use static, allowlisted adapters only

Load adapters when the server starts, from directories the CLI installed and verified. Requests pick an adapter by name with the model field. Nothing is loaded or unloaded while the server runs.

retoque verify <slug>@<version>

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 \
  --enable-lora \
  --max-lora-rank 16 \
  --lora-modules <slug>=/srv/retoque/patches/<slug>/<version>

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "<slug>",
    "temperature": 0,
    "max_tokens": 128,
    "messages": [
      {"role": "system", "content": "<the system prompt the patch was evaluated with>"},
      {"role": "user", "content": "Today is 2026-03-02 (Monday).\nPay Halcyon Realty $4,200.00 for the office rent, due next Friday."}
    ]
  }'

• Pin --revision to the full commit in the manifest. A branch name can move.
• Set --max-lora-rank to at least the adapter.rank in the manifest.
• Mount the install directory read-only in the container. The CLI already makes the files read-only on disk.
• To serve the base model without the patch, send the base model name in model. That is the vLLM equivalent of retoque disable.

Never expose dynamic LoRA endpoints

• Leave VLLM_ALLOW_RUNTIME_LORA_UPDATING unset on any public or shared server.
• If you need hot swapping, keep the management endpoints on a private, authenticated network, and only load directories that passed retoque verify.
• Put the inference API itself behind your own authentication. vLLM has no per-patch license checks.

Licensing when you serve a patch

Serving a patch to other people is covered only if the version's license terms allow it. Check the commercial scope on the patch page before you deploy. A free developer license does not automatically cover a hosted service. See Licenses and entitlements.