Qwen3.8-Flash-Next on 2× CMP 170HX

Qwen3.8-Flash-Next (W4A16) on two NVIDIA CMP 170HX cards, pipeline-parallel with vLLM: 234 tok/s for one user (code), 827 tok/s across eight, 6,200–7,400 tok/s prefill from 4k to 64k prompts, the full 262,144-token window, image and video input, and HumanEval 157/164, GSM8K 1,272/1,319. Measured on PCIe Gen2 x4 links.

Serves Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with upstream vLLM and the vllm-cmp170hx plugin in a container: an OpenAI-compatible API with text, image and video input, native MTP speculative decoding, reasoning and tool calls. The ~102 GB N-gram (PLE) table stays on the NVMe drive and is read through the page cache; everything else is on the two GPUs.

git clone https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next
cd cmp170hx-qwen38-flash-next
./download.sh     # checkpoint at the pinned revision (~170 GB)
./start.sh        # tunes the host (sudo), starts the container, waits for /health

The API listens on 127.0.0.1:8000 as model qwen3.8-flash-next-w4a16. ./stop.sh stops it. Settings go in…