Repository navigation
[FT] Log progress bar on main process #662
Description
Issue encountered
During evaluation one sees repeated progress bars that don't convey the global progress towards an eval being finished, e.g.
Processed prompts: 0%| | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s] [repeated 7x across cluster]
Processed prompts: 25%|██▌ | 1/4 [00:14<00:43, 14.53s/it, est. speed input: 15.14 toks/s, output: 221.23 toks/s] [repeated 3x across cluster]
Processed prompts: 50%|█████ | 2/4 [00:22<00:23, 11.99s/it, est. speed input: 14.81 toks/s, output: 299.95 toks/s] [repeated 2x across cluster]
Processed prompts: 50%|█████ | 2/4 [00:30<00:27, 13.71s/it, est. speed input: 9.25 toks/s, output: 389.36 toks/s] [repeated 4x across cluster]
Processed prompts: 100%|██████████| 4/4 [00:35<00:00, 8.89s/it, est. speed input: 27.27 toks/s, output: 577.98 toks/s]
(run_inference_one_model pid=3713070) [rank0]:[W407 09:31:09.544209083 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Processed prompts: 33%|███▎ | 1/3 [00:43<01:27, 43.61s/it, est. speed input: 3.26 toks/s, output: 192.65 toks/s]
Processed prompts: 75%|███████▌ | 3/4 [01:17<00:30, 30.61s/it, est. speed input: 6.71 toks/s, output: 244.39 toks/s]
Processed prompts: 67%|██████▋ | 2/3 [01:24<00:42, 42.04s/it, est. speed input: 4.23 toks/s, output: 229.77 toks/s]
Processed prompts: 50%|█████ | 2/4 [01:48<02:06, 63.09s/it, est. speed input: 2.57 toks/s, output: 126.10 toks/s]
Processed prompts: 100%|██████████| 3/3 [01:52<00:00, 37.62s/it, est. speed input: 4.68 toks/s, output: 286.81 toks/s]
(run_inference_one_model pid=3713065) [rank0]:[W407 09:32:27.439066098 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Processed prompts: 75%|███████▌ | 3/4 [01:58<00:47, 47.32s/it, est. speed input: 4.58 toks/s, output: 212.09 toks/s]
Processed prompts: 50%|█████ | 2/4 [02:22<02:40, 80.26s/it, est. speed input: 3.26 toks/s, output: 130.89 toks/s]
Processed prompts: 100%|██████████| 4/4 [02:42<00:00, 40.55s/it, est. speed input: 4.09 toks/s, output: 215.71 toks/s]
(run_inference_one_model pid=3713080) [rank0]:[W407 09:33:18.025165983 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Processed prompts: 75%|███████▌ | 3/4 [02:46<01:00, 60.80s/it, est. speed input: 2.72 toks/s, output: 180.25 toks/s]
Processed prompts: 100%|██████████| 4/4 [02:50<00:00, 42.71s/it, est. speed input: 4.44 toks/s, output: 298.07 toks/s]
(run_inference_one_model pid=3713072) [rank0]:[W407 09:33:24.914485162 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Processed prompts: 75%|███████▌ | 3/4 [02:48<00:55, 55.45s/it, est. speed input: 3.69 toks/s, output: 206.29 toks/s]
Processed prompts: 25%|██▌ | 1/4 [02:59<08:57, 179.08s/it, est. speed input: 0.99 toks/s, output: 93.36 toks/s]
Processed prompts: 67%|██████▋ | 2/3 [03:45<02:10, 130.81s/it, est. speed input: 1.66 toks/s, output: 97.46 toks/s]
Processed prompts: 50%|█████ | 2/4 [04:18<04:00, 120.37s/it, est. speed input: 1.19 toks/s, output: 148.48 toks/s]
Processed prompts: 100%|██████████| 4/4 [04:34<00:00, 68.68s/it, est. speed input: 2.64 toks/s, output: 176.78 toks/s]
This makes it hard to know how long an eval will take in total.
Solution/Feature
I think it should be possible to solve this by having the progress bar log iterations on the main process.
Possible alternatives
If not possible to log on the main process, providing some additional information in the progress bar description that allows users to reconstruct the global progress would be good.
Activity
Honorllq commentedon Sep 16, 2026
Would you accept a scoped follow-up to #1340 for this issue? That PR was closed unmerged by its author; current main still lets each Ray worker display a vLLM progress bar while the driver blocks on ray.get.
Proposed scope: suppress the worker bars and report aggregate progress on the driver as worker batches finish, preserving request/result ordering and existing inference settings. The display would explicitly describe batch-completion progress, not token-level progress or a precise ETA.
Validation plan: focused tests for out-of-order completion, uneven/empty shards, worker errors, and unchanged output ordering, avoiding duplication of #1377's general backend tests; then a real multi-GPU data-parallel smoke test before any PR (GPU access would be needed).
Main risks are misleading progress granularity and regressions in Ray cleanup or result ordering. Is this limited scope still useful? I have not started implementation.
Steeve-Crypto commentedon Oct 7, 2026
I'll take this. Plan: suppress per-worker vLLM progress bars under Ray and show a single driver-side bar (follow-up to the approach in #1340). Opening a PR.