feat: add NPU device support for embedded-python model executor.

[yingxudeng]

Description

Related Issues

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactor
• Documentation
• Test
• Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

• The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit or an equivalent command.
• I have installed the hooks with pre-commit install.
• I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

• I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
• I have rebased this PR onto the latest main branch.

Build and Test Coverage

• Tests have been added or updated as needed.
• CUDA: python setup.py build test has passed on a CUDA machine.
• NPU: python setup.py build test has passed on an NPU machine.
• MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[yingxudeng]

TP2 End-to-End Verification

Branch: feat/npu-python-graph-tp-post4 @ 699e839e
Container: dyx-xllm-cann9-main-0723
Model: Qwen3-4B (BFloat16)
Devices: npu:2, npu:3 (via ASCEND_RT_VISIBLE_DEVICES=2,3)

Environment

Component	Version
torch_npu	2.9.0.post4+git44b5898
torch	2.9.0+cpu
CANN	9.0
libtorch_npu.so	71,305,432 bytes (post4)

Startup Command

ASCEND_RT_VISIBLE_DEVICES=2,3 \
LD_LIBRARY_PATH=/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib:$LD_LIBRARY_PATH \
./build/lib.linux-aarch64-cpython-311/xllm/xllm \
  --model /export/home/models/Qwen3-4B \
  --port 8199 --host 0.0.0.0 \
  --max_seqs_per_batch 4 \
  --python_graph_backend eager \
  --model_impl python

Service Logs (key lines)

Forced npu_kernel_backend=TORCH for python model_impl
Creating engine with devices: npu:0,npu:1
Multi-node serving world_size = 2, each_node_ranks = 2, tp_size = 2
Application startup complete.

Inference Results

Test	Input	Output	Status
Math	2+3等于几？	5 (with think)	PASS
Chinese	中国的首都是哪里？	北京 (with think)	PASS
Code	def fibonacci(n):	correct implementation	PASS
English	Capital of France?	Paris (with think)	PASS

[yingxudeng]

## NPU Python Graph TP=2 精度验证报告

### 环境

| 项目 | 值 |
|------|-----|
| 容器 | `dyx-xllm-cann9-main-0723` (from `dyx-xllm-cann9-main:backup-20260722`) |
| CANN | 9.0.0 |
| torch_npu | 2.9.0.post4+git44b5898 |
| 模型 | Qwen3-4B |
| TP | 2 (多进程, nnodes=2) |
| 设备 | NPU 14, 15 |
| Commit | aeb03a24 |

### 复现步骤

#### 1. 编译

```bash
docker exec -w /export/home/dengyingxu1/projects/xllm-feat-qwen3-npu-python-graph \
  dyx-xllm-cann9-main-0723 bash -lc 'python setup.py build --device npu'

编译时 setup.py 会自动 pip install torch_npu post4，链接仍使用 /usr/local/libtorch_npu/lib/（post2），无需修改。

2. 启动 TP=2 服务 (多进程)

CONTAINER=dyx-xllm-cann9-main-0723
CHECKOUT=/export/home/dengyingxu1/projects/xllm-feat-qwen3-npu-python-graph
MODEL=/export/home/models/Qwen3-4B
XLLM_BIN="./build/lib.linux-aarch64-cpython-311/xllm/xllm"

# ⚠️ 关键：Python executor 需要 torch_npu >= post4 的 .so（含 npu_extension_functions 符号）。
# 系统 /usr/local/libtorch_npu/lib/ 是 post2（ATB backend 使用），不能替换。
# 通过 LD_LIBRARY_PATH 前置 pip 安装的 post4 路径，让 ld.so 优先加载 post4 版本。
# 这样 ATB backend 启动时不加此变量，仍使用 post2，互不影响。
TORCH_NPU_LIB="/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib"

# Rank 0
docker exec $CONTAINER bash -c "
  export ASCEND_RT_VISIBLE_DEVICES=14,15
  export LD_LIBRARY_PATH=$TORCH_NPU_LIB:\$LD_LIBRARY_PATH
  cd $CHECKOUT
  nohup $XLLM_BIN \
    --model $MODEL --port 8199 --host 0.0.0.0 \
    --max_seqs_per_batch 4 \
    --model_impl python --python_graph_backend off \
    --nnodes 2 --node_rank 0 --master_node_addr=127.0.0.1:19888 \
    --enable_prefix_cache=false --enable_chunked_prefill=false \
    > /tmp/xllm_tp2_rank0.log 2>&1 &"

# Rank 1
docker exec $CONTAINER bash -c "
  export ASCEND_RT_VISIBLE_DEVICES=14,15
  export LD_LIBRARY_PATH=$TORCH_NPU_LIB:\$LD_LIBRARY_PATH
  cd $CHECKOUT
  nohup $XLLM_BIN \
    --model $MODEL --port 8200 --host 0.0.0.0 \
    --max_seqs_per_batch 4 \
    --model_impl python --python_graph_backend off \
    --nnodes 2 --node_rank 1 --master_node_addr=127.0.0.1:19888 \
    --enable_prefix_cache=false --enable_chunked_prefill=false \
    > /tmp/xllm_tp2_rank1.log 2>&1 &"

3. 精度评测

docker exec $CONTAINER bash -c "
  evalscope eval \
    --model Qwen3-4B \
    --api-url http://localhost:8199/v1 \
    --api-key dummy \
    --datasets gsm8k \
    --limit 10 \
    --eval-batch-size 4 \
    --timeout 180 \
    --work-dir /tmp/evalscope_tp2"

结果

┌──────────┬───────────┬──────────┬──────────┬───────┬─────────┐
│ Model    │ Dataset   │ Metric   │ Subset   │   Num │   Score │
├──────────┼───────────┼──────────┼──────────┼───────┼─────────┤
│ Qwen3-4B │ gsm8k     │ mean_acc │ main     │    10 │       1 │
└──────────┴───────────┴──────────┴──────────┴───────┴─────────┘

Model     Dataset      Num    Avg Lat    Avg Thpt    Avg In    Avg Out
                                  (s)     (tok/s)       Tok        Tok
--------  ---------  -----  ---------  ----------  --------  ---------
Qwen3-4B  gsm8k         10     48.017       32.93     662.6     1581.1

~~

关键说明：LD_LIBRARY_PATH 机制

Python executor 依赖 torch_npu post4 的 npu_extension_functions 符号（post4 新增）。但容器中存在两份 libtorch_npu.so：

路径	版本	用途
/usr/local/libtorch_npu/lib/	post2	xllm 编译链接、ATB backend 运行时
.../site-packages/torch_npu/lib/	post4	pip 安装，Python executor 需要

策略：不替换系统 .so，启动时 LD_LIBRARY_PATH 切换。

• Python executor 启动：LD_LIBRARY_PATH=<post4路径>:$LD_LIBRARY_PATH → ld.so 优先加载 post4
• ATB backend 启动：不设此变量 → 默认加载 /usr/local/libtorch_npu/lib/ 的 post2

两条路径互不干扰。长期方案：验证 post4 对 ATB 完全向后兼容后，统一升级系统 .so，删除 LD_LIBRARY_PATH 覆盖。

其他注意事项

• Bool flag 必须用 = 语法（--enable_prefix_cache=false），不能 --flag false（gflags 将前者解析为 true）
• npu_kernel_backend 无需手动设置，代码自动检测 model_impl=python 后 force TORCH
• ASCEND_RT_VISIBLE_DEVICES 两个 rank 都设为全部 TP 卡（14,15），代码内部按 node_rank 索引选卡
  ~~]

[yingxudeng]

LD_LIBRARY_PATH 不用管了，已经统一都用 whl 包下面的 .so 了