MRV2 beam-search scheduler and sampler plugin for vLLM V1.
This package provides:
vllm_beam_search.scheduler.BeamSearchScheduler- plugin-local sequence, token, and KV admission for complete beam groups
- an MRV2 custom sampler wrapper installed through a plugin-local
ModelStatehook - plugin-local runtime hooks for MRV2 worker history rewrites
The current production path targets MRV2 generate models with async scheduling.
The sampler hook is model-state generic; BART-family models still need the
companion vllm-bart-plugin for encoder-decoder model support.
The plugin does not require a vLLM fork or source patch. It carries explicit
plugin-local scheduler implementations for vLLM 0.24.0, 0.26.0, the tested
0.26.1 development build, 0.28.0, 0.31.0, and 0.33.0. Startup fails closed on
an unsupported scheduler.
Each vendored scheduler has an adjacent .diff recording its exact changes
from the hashed upstream Scheduler.schedule; the test suite verifies both.
The Git repository, installable Python distribution, and PyPI project are named
vllm-beam-search-plugin; the import package is vllm_beam_search.
For BART-family encoder-decoder serving, see
BART_BEAM_SEARCH.md.
Install the published distribution from PyPI with an exact version pin:
uv pip install 'vllm-beam-search-plugin==0.1.6'The plugin metadata constrains its tested NumPy, PyTorch, and Triton API ranges. vLLM remains the owner of their accelerator-specific builds. In a prebuilt RHAII 3.5 image, preserve the image runtime and install only the pure-Python plugin:
uv pip install --no-deps \
'vllm-beam-search-plugin==0.1.6'The vLLM 0.24.0 and 0.28.0 paths have been unit-, correctness-, concurrency-,
and sustained-memory-tested. The vLLM 0.31.0 path has been unit-, correctness-,
and concurrency-tested with decoder-only models. The vLLM 0.33.0 path has been
unit-, correctness-, concurrency-, and abort-tested with decoder-only models and
BART. Each selects its explicit vendored scheduler directly; the plugin does not
use inspect.getsource, source-text matching, or runtime exec.
For stress tooling:
uv pip install 'vllm-beam-search-plugin[stress]==0.1.6'MODEL=${MODEL:-meta-llama/Meta-Llama-3-8B-Instruct}
SERVED_MODEL=${SERVED_MODEL:-llama3-8b}
CUDA_VISIBLE_DEVICES=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve "${MODEL}" \
--served-model-name "${SERVED_MODEL}" \
--dtype bfloat16 \
--port 8005 \
--scheduler-cls vllm_beam_search.scheduler.BeamSearchScheduler{
"model": "llama3-8b",
"prompt": "Write a concise summary of why beam search is useful:",
"max_tokens": 128,
"temperature": 0,
"add_special_tokens": false,
"vllm_xargs": {
"beam_width": 4,
"no_repeat_ngram_size": 3
}
}Run unit tests:
uv run --with pytest python -m pytest tests -qRun sustained stress plus memory sampling against a running server:
vllm-beam-stress \
--base-url http://localhost:8005 \
--model llama3-8b \
--rounds 100 \
--requests-per-round 32 \
--concurrency 64 \
--abort-rounds 3The stress tool writes CSV samples with request count, RSS, and GPU memory.
VLLM_BEAM_GROUP_STATE_CAPACITYcontrols GPU beam-state pool capacity.VLLM_BEAM_TRANSITION_BUFFER_SLOTScontrols async transition buffer slots.