[llama3.1 8b][mxfp8] inference error on sm120 when using FLASHINFER attention backend

[XuehaoSun]

--iters 200 --scheme MXFP8 --static_kv_dtype fp8 --static_attention_dtype fp8 --format llm_compressor --device_map 0

vllm serve Llama-3.1-8B-Instruct-mxfp-w8g32 --port 7777 --host localhost --trust-remote-code --dtype bfloat16 --tensor_parallel_size 2 --max-model-len 4096 --max-num-seqs 32 --gpu-memory-utilization 0.85 --disable-custom-all-reduce

quant log: log
eval log: log

--- Test 1/4: Number comparison ---
Question: Which of these two numbers is larger: 9.11 or 9.8? Please answer directly with the larger number.
Model response: href na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na na

--- Test 2/4: Simple math calculation ---
Question: Calculate 9.9 - 9.11? Please answer directly with the result.
Model response: 百度 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00 2020-01-01 00:00:00

--- Test 3/4: Basic knowledge ---
Question: What is the capital of France? Please answer directly.
Model response: 百度 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00 2021-01-01 00:00:00

--- Test 4/4: Simple logical reasoning ---
Question: I have a pen, an apple, and a bicycle. Which one can be used for writing? Please answer directly.
Model response: in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom, and the  in the United States, and the  in the United Kingdom

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.003|±  |0.0015|
|     |       |strict-match    |     5|exact_match|↑  |0.000|±  |0.0000|

[XuehaoSun]

Using the triton-attn backend works normally.

vllm serve Llama-3.1-8B-Instruct-mxfp-w8g32 --port 7777 --host localhost --trust-remote-code --dtype bfloat16 \
    --tensor_parallel_size 2 --max-model-len 4096 --max-num-seqs 32 --gpu-memory-utilization 0.85 --disable-custom-all-reduce \
    --attention-backend TRITON_ATTN --kv-cache-dtype fp8

--- Test 1/4: Number comparison ---
Question: Which of these two numbers is larger: 9.11 or 9.8? Please answer directly with the larger number.
Model response: 9.8

--- Test 2/4: Simple math calculation ---
Question: Calculate 9.9 - 9.11? Please answer directly with the result.
Model response: -0.2

--- Test 3/4: Basic knowledge ---
Question: What is the capital of France? Please answer directly.
Model response: Paris.

--- Test 4/4: Simple logical reasoning ---
Question: I have a pen, an apple, and a bicycle. Which one can be used for writing? Please answer directly.
Model response: The pen.

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8158|±  |0.0107|
|     |       |strict-match    |     5|exact_match|↑  |0.7908|±  |0.0112|