W0923 17:22:02.312000 1 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:02.329000 1 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
(APIServer pid=1) [transformers] The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
W0923 17:22:20.295000 241 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:20.307000 241 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:26.140000 264 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:26.140000 263 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:26.153000 264 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0923 17:22:26.153000 263 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   0% Completed | 0/120 [00:00<?, ?it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   1% Completed | 1/120 [00:00<01:19,  1.49it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   2% Completed | 2/120 [00:00<00:43,  2.71it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   2% Completed | 3/120 [00:01<00:34,  3.39it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   3% Completed | 4/120 [00:01<00:36,  3.22it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   4% Completed | 5/120 [00:01<00:28,  3.99it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   5% Completed | 6/120 [00:01<00:28,  3.95it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   6% Completed | 7/120 [00:02<00:31,  3.58it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   7% Completed | 8/120 [00:02<00:26,  4.22it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   8% Completed | 9/120 [00:02<00:28,  3.85it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   8% Completed | 10/120 [00:02<00:32,  3.36it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:   9% Completed | 11/120 [00:03<00:28,  3.83it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  10% Completed | 12/120 [00:03<00:32,  3.29it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  11% Completed | 13/120 [00:03<00:28,  3.78it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  12% Completed | 14/120 [00:03<00:25,  4.08it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  12% Completed | 15/120 [00:04<00:35,  2.94it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  13% Completed | 16/120 [00:04<00:29,  3.56it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  14% Completed | 17/120 [00:04<00:27,  3.68it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  15% Completed | 18/120 [00:05<00:30,  3.36it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  16% Completed | 19/120 [00:05<00:25,  3.94it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  17% Completed | 20/120 [00:05<00:28,  3.57it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  18% Completed | 21/120 [00:05<00:26,  3.67it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  18% Completed | 22/120 [00:06<00:23,  4.25it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  19% Completed | 23/120 [00:06<00:28,  3.42it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  20% Completed | 24/120 [00:06<00:27,  3.51it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  21% Completed | 25/120 [00:07<00:24,  3.82it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  22% Completed | 26/120 [00:07<00:28,  3.30it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  22% Completed | 27/120 [00:07<00:23,  3.89it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  23% Completed | 28/120 [00:07<00:22,  4.13it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  24% Completed | 29/120 [00:08<00:27,  3.34it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  25% Completed | 30/120 [00:08<00:22,  3.94it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  26% Completed | 31/120 [00:08<00:22,  3.93it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  27% Completed | 32/120 [00:08<00:24,  3.62it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  28% Completed | 33/120 [00:09<00:20,  4.17it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  28% Completed | 34/120 [00:09<00:24,  3.47it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  29% Completed | 35/120 [00:09<00:23,  3.68it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  30% Completed | 36/120 [00:09<00:20,  4.12it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  31% Completed | 37/120 [00:10<00:24,  3.34it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  32% Completed | 38/120 [00:10<00:22,  3.64it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  32% Completed | 39/120 [00:10<00:21,  3.80it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  33% Completed | 40/120 [00:11<00:25,  3.19it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  34% Completed | 41/120 [00:11<00:21,  3.73it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  35% Completed | 42/120 [00:11<00:21,  3.56it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  36% Completed | 43/120 [00:12<00:24,  3.09it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  37% Completed | 44/120 [00:12<00:21,  3.57it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  38% Completed | 45/120 [00:12<00:30,  2.48it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  38% Completed | 46/120 [00:13<00:30,  2.45it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  39% Completed | 47/120 [00:13<00:24,  2.92it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  40% Completed | 48/120 [00:14<00:33,  2.13it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  41% Completed | 49/120 [00:14<00:28,  2.53it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  42% Completed | 50/120 [00:14<00:25,  2.78it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  42% Completed | 51/120 [00:15<00:27,  2.47it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  43% Completed | 52/120 [00:15<00:22,  3.02it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  44% Completed | 53/120 [00:15<00:21,  3.10it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  45% Completed | 54/120 [00:16<00:24,  2.71it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  46% Completed | 55/120 [00:16<00:20,  3.21it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  47% Completed | 56/120 [00:16<00:21,  2.99it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  48% Completed | 57/120 [00:17<00:23,  2.71it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  48% Completed | 58/120 [00:17<00:18,  3.26it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  49% Completed | 59/120 [00:17<00:22,  2.69it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  50% Completed | 60/120 [00:18<00:19,  3.01it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  51% Completed | 61/120 [00:18<00:17,  3.45it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  52% Completed | 62/120 [00:18<00:19,  2.91it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  52% Completed | 63/120 [00:19<00:16,  3.46it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  53% Completed | 64/120 [00:19<00:15,  3.51it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  54% Completed | 65/120 [00:19<00:18,  2.97it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  55% Completed | 66/120 [00:19<00:15,  3.54it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  56% Completed | 67/120 [00:20<00:15,  3.36it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  57% Completed | 68/120 [00:20<00:16,  3.21it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  57% Completed | 69/120 [00:20<00:13,  3.70it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  58% Completed | 70/120 [00:21<00:16,  3.05it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  59% Completed | 71/120 [00:21<00:15,  3.22it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  60% Completed | 72/120 [00:21<00:13,  3.69it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  61% Completed | 73/120 [00:22<00:15,  3.12it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  62% Completed | 74/120 [00:22<00:12,  3.61it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  62% Completed | 75/120 [00:22<00:12,  3.69it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  63% Completed | 76/120 [00:23<00:14,  3.09it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  64% Completed | 77/120 [00:23<00:11,  3.69it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  65% Completed | 78/120 [00:23<00:12,  3.41it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  66% Completed | 79/120 [00:23<00:13,  3.12it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  67% Completed | 80/120 [00:24<00:10,  3.74it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  68% Completed | 81/120 [00:24<00:13,  2.87it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  68% Completed | 82/120 [00:24<00:13,  2.90it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  69% Completed | 83/120 [00:25<00:11,  3.28it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  70% Completed | 84/120 [00:25<00:13,  2.74it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  71% Completed | 85/120 [00:25<00:10,  3.23it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  72% Completed | 86/120 [00:26<00:09,  3.43it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  72% Completed | 87/120 [00:26<00:12,  2.64it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  73% Completed | 88/120 [00:26<00:10,  2.93it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  74% Completed | 89/120 [00:27<00:11,  2.62it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  75% Completed | 90/120 [00:27<00:12,  2.35it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  76% Completed | 91/120 [00:28<00:10,  2.90it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  77% Completed | 92/120 [00:28<00:10,  2.59it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  78% Completed | 93/120 [00:28<00:10,  2.63it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  78% Completed | 94/120 [00:29<00:08,  3.09it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  79% Completed | 95/120 [00:29<00:10,  2.40it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  80% Completed | 96/120 [00:29<00:08,  2.98it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  81% Completed | 97/120 [00:30<00:07,  3.16it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  82% Completed | 98/120 [00:30<00:07,  2.76it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  82% Completed | 99/120 [00:30<00:06,  3.29it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  83% Completed | 100/120 [00:31<00:06,  2.91it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  84% Completed | 101/120 [00:31<00:06,  2.75it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  85% Completed | 102/120 [00:31<00:05,  3.32it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  86% Completed | 103/120 [00:32<00:05,  2.97it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  87% Completed | 104/120 [00:32<00:04,  3.32it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  88% Completed | 106/120 [00:32<00:02,  4.76it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  89% Completed | 107/120 [00:32<00:02,  4.64it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  90% Completed | 108/120 [00:33<00:02,  4.50it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  91% Completed | 109/120 [00:33<00:03,  3.44it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  92% Completed | 110/120 [00:33<00:02,  4.14it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  92% Completed | 111/120 [00:34<00:02,  3.77it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  93% Completed | 112/120 [00:34<00:02,  3.17it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  94% Completed | 113/120 [00:34<00:01,  3.77it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  95% Completed | 114/120 [00:35<00:01,  3.22it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  96% Completed | 115/120 [00:35<00:01,  3.24it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  97% Completed | 116/120 [00:35<00:01,  3.79it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  98% Completed | 117/120 [00:35<00:01,  2.98it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  98% Completed | 118/120 [00:36<00:00,  3.23it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards:  99% Completed | 119/120 [00:36<00:00,  3.51it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards: 100% Completed | 120/120 [00:36<00:00,  3.21it/s]
(Worker_TP0_EP0 pid=263)
Loading safetensors checkpoint shards: 100% Completed | 120/120 [00:36<00:00,  3.26it/s]
(Worker_TP0_EP0 pid=263)
(Worker_TP1_EP1 pid=264) /usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
(Worker_TP1_EP1 pid=264)   warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
(Worker_TP0_EP0 pid=263) /usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
(Worker_TP0_EP0 pid=263)   warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
[17:23:27] : Warning: T.vectorized loop over `i_hci` with extent 4 is lowered as a serial loop because TileLang could not find a valid vectorization plan. Scalar accumulator updates inside the loop are a common cause; move reductions to T.unroll or T.serial if this is intended.
[17:23:27] : Warning: T.vectorized loop over `i_hci` with extent 4 is lowered as a serial loop because TileLang could not find a valid vectorization plan. Scalar accumulator updates inside the loop are a common cause; move reductions to T.unroll or T.serial if this is intended.
(Worker_TP0_EP0 pid=263) /usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
(Worker_TP0_EP0 pid=263)   warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
(Worker_TP1_EP1 pid=264) /usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
(Worker_TP1_EP1 pid=264)   warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
(Worker_TP0_EP0 pid=263)
Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:00<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:00<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:05<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:05<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:06<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:06<?, ?it/s]


Profiling CUDA graph memory (PIECEWISE):   0%|          | 0/4 [00:10<?, ?it/s]
Profiling CUDA graph memory (PIECEWISE):  25%|██▌       | 1/4 [00:11<00:35, 11.92s/it]


Profiling CUDA graph memory (PIECEWISE):  25%|██▌       | 1/4 [00:12<00:35, 11.92s/it]


Profiling CUDA graph memory (PIECEWISE):  25%|██▌       | 1/4 [00:13<00:35, 11.92s/it]


Profiling CUDA graph memory (PIECEWISE):  25%|██▌       | 1/4 [00:13<00:35, 11.92s/it]


Profiling CUDA graph memory (PIECEWISE):  25%|██▌       | 1/4 [00:17<00:35, 11.92s/it]
Profiling CUDA graph memory (PIECEWISE):  50%|█████     | 2/4 [00:19<00:19,  9.64s/it]
Profiling CUDA graph memory (PIECEWISE):  75%|███████▌  | 3/4 [00:20<00:05,  5.56s/it]
Profiling CUDA graph memory (PIECEWISE): 100%|██████████| 4/4 [00:21<00:00,  3.86s/it]
Profiling CUDA graph memory (PIECEWISE): 100%|██████████| 4/4 [00:21<00:00,  5.48s/it]
(Worker_TP0_EP0 pid=263)
Profiling CUDA graph memory (FULL):   0%|          | 0/4 [00:00<?, ?it/s]
Profiling CUDA graph memory (FULL):  25%|██▌       | 1/4 [00:00<00:00,  6.17it/s]
Profiling CUDA graph memory (FULL):  50%|█████     | 2/4 [00:00<00:00,  4.97it/s]
Profiling CUDA graph memory (FULL):  75%|███████▌  | 3/4 [00:00<00:00,  5.60it/s]
Profiling CUDA graph memory (FULL): 100%|██████████| 4/4 [00:00<00:00,  6.07it/s]
Profiling CUDA graph memory (FULL): 100%|██████████| 4/4 [00:00<00:00,  5.83it/s]
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:44,306 - INFO - autotuner.py:809 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:44,306 - INFO - autotuner.py:809 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:44,507 - INFO - autotuner.py:832 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:44,507 - INFO - autotuner.py:832 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:44,522 - INFO - autotuner.py:2264 - flashinfer.jit: [Autotuner]: Saved 0 configs to /cache/jit/cu133-torch213-vllm174c789e09-b12x12c426322c-lmcachee045d729bc/flashinfer-autotune/31bab91f6e82675aae8970504d4f53ef2fbe8ea4fcc514054abed69eb1895c59/autotune_configs.json (0 new, 0 from previous config)
(Worker_TP0_EP0 pid=263)
Capturing CUDA graphs (PIECEWISE):   0%|          | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):  25%|██▌       | 1/4 [00:00<00:02,  1.50it/s]
Capturing CUDA graphs (PIECEWISE):  50%|█████     | 2/4 [00:01<00:01,  1.54it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▌  | 3/4 [00:01<00:00,  1.56it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 4/4 [00:02<00:00,  1.57it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 4/4 [00:02<00:00,  1.56it/s]
(Worker_TP0_EP0 pid=263)
Capturing CUDA graphs (FULL):   0%|          | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL):  25%|██▌       | 1/4 [00:00<00:00,  6.86it/s]
Capturing CUDA graphs (FULL):  50%|█████     | 2/4 [00:00<00:00,  6.86it/s]
Capturing CUDA graphs (FULL):  75%|███████▌  | 3/4 [00:00<00:00,  6.89it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 4/4 [00:00<00:00,  6.92it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 4/4 [00:00<00:00,  6.90it/s]
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.
containment-passed
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.1.dev20111+g7f1e92bec.d20260827
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]   █▄█▀ █     █     █     █  model   /root/.cache/huggingface
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:345]
(APIServer pid=1) INFO 09-23 17:22:05 [api_utils.py:273] non-default args: {'model_tag': '/root/.cache/huggingface', 'chat_template': '/opt/glm53/chat_template.multimodal.jinja', 'enable_auto_tool_choice': True, 'tool_call_parser': 'glm47', 'host': '0.0.0.0', 'port': 8002, 'model': '/root/.cache/huggingface', 'dtype': 'bfloat16', 'max_model_len': 327680, 'served_model_name': ['glm53-flash-exl3-runtime-dcp1-candidate'], 'generation_config': '/root/.cache/huggingface', 'load_format': 'safetensors', 'attention_backend': 'FLASHINFER_MLA_SPARSE_SM120', 'reasoning_parser': 'glm45', 'tensor_parallel_size': 2, 'dcp_comm_backend': 'a2a', 'enable_expert_parallel': True, 'disable_custom_all_reduce': True, 'gpu_memory_utilization': 0.97, 'kv_cache_dtype': 'fp8_ds_mla', 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 8, 'video': 0}, 'max_num_batched_tokens': 2048, 'max_num_seqs': 4, 'enable_chunked_prefill': True, 'moe_backend': 'b12x', 'reasoning_config': ReasoningConfig(reasoning_parser='', reasoning_start_str='', reasoning_end_str='')}
(APIServer pid=1) WARNING 09-23 17:22:05 [envs.py:2544] Unknown vLLM environment variable detected: VLLM_B12X_GLM_NOPE_NVFP4
(APIServer pid=1) INFO 09-23 17:22:13 [model.py:680] Resolved architecture: Glm5NextForConditionalGeneration
(APIServer pid=1) INFO 09-23 17:22:13 [model.py:1975] Using max model len 327680
(APIServer pid=1) INFO 09-23 17:22:15 [cache.py:283] Using fp8_ds_mla data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1) INFO 09-23 17:22:15 [config.py:598] Mamba cache mode is set to 'align' for Glm5NextForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 09-23 17:22:15 [exl3.py:1284] GLM-5.3 routed-only EXL3: streaming unsliced K4 experts into TP2 B12X slabs for main layers 3..44 and MTP layer 45
(APIServer pid=1) WARNING 09-23 17:22:15 [vllm.py:145] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1 for this architecture, disabling VLLM_USE_AOT_COMPILE=1.
(APIServer pid=1) INFO 09-23 17:22:15 [vllm.py:149] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out.
(APIServer pid=1) INFO 09-23 17:22:15 [kernel.py:310] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=1) INFO 09-23 17:22:15 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=241) INFO 09-23 17:22:22 [core.py:121] Initializing a V1 LLM engine (v0.1.dev20111+g7f1e92bec.d20260827) with config: model='/root/.cache/huggingface', speculative_config=None, tokenizer='/root/.cache/huggingface', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=327680, download_dir=None, load_format=safetensors, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=exl3, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_ds_mla, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='glm45', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=glm53-flash-exl3-runtime-dcp1-candidate, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='b12x', linear_backend='auto')
(EngineCore pid=241) INFO 09-23 17:22:22 [multiproc_executor.py:165] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=100.64.0.10 (local), world_size=2, local_world_size=2
(Worker pid=263) INFO 09-23 17:22:31 [parallel_state.py:1938] world_size=2 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_0da880958d2a4578819a81d6824758ba backend=nccl
(Worker pid=264) INFO 09-23 17:22:31 [parallel_state.py:1938] world_size=2 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_0da880958d2a4578819a81d6824758ba backend=nccl
(Worker pid=264) INFO 09-23 17:22:31 [nccl.py:25] Found nccl from environment variable VLLM_NCCL_SO_PATH=/opt/local-inference/nccl/lib/libnccl.so.2.31.2
(Worker pid=263) INFO 09-23 17:22:31 [nccl.py:25] Found nccl from environment variable VLLM_NCCL_SO_PATH=/opt/local-inference/nccl/lib/libnccl.so.2.31.2
(Worker pid=263) INFO 09-23 17:22:31 [pynccl.py:113] vLLM is using nccl==2.31.2
(Worker pid=263) WARNING 09-23 17:22:31 [symm_mem.py:101] SymmMemCommunicator: native P2P atomics are not supported between devices [0, 1], communicator is not available.
(Worker pid=264) WARNING 09-23 17:22:31 [symm_mem.py:101] SymmMemCommunicator: native P2P atomics are not supported between devices [0, 1], communicator is not available.
(Worker pid=263) INFO 09-23 17:22:31 [cuda_communicator.py:274] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=263) INFO 09-23 17:22:31 [nccl.py:25] Found nccl from environment variable VLLM_NCCL_SO_PATH=/opt/local-inference/nccl/lib/libnccl.so.2.31.2
(Worker pid=264) INFO 09-23 17:22:31 [nccl.py:25] Found nccl from environment variable VLLM_NCCL_SO_PATH=/opt/local-inference/nccl/lib/libnccl.so.2.31.2
(Worker pid=263) INFO 09-23 17:22:31 [cuda_communicator.py:274] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=263) INFO 09-23 17:22:31 [parallel_state.py:2344] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=263) INFO 09-23 17:22:31 [gpu_worker.py:414] Using V2 Model Runner
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [model_runner.py:447] Loading model from scratch...
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [cuda.py:589] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [mm_encoder_attention.py:375] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:22:32 [kernel.py:310] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [kernel.py:310] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(Worker_TP1_EP1 pid=264) INFO 09-23 17:22:32 [cuda.py:470] Using AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120 backend.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [cuda.py:470] Using AttentionBackendEnum.FLASHINFER_MLA_SPARSE_SM120 backend.
(Worker_TP0_EP0 pid=263) WARNING 09-23 17:22:32 [mla_attention.py:844] Sparse MLA layer has no dense-MHA prefill path; using the top-k MQA path only.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [deep_gemm.py:207] DeepGEMM PDL enabled on deep_gemm.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [deep_gemm.py:125] DeepGEMM E8M0 enabled on current platform.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [expert_map_manager.py:245] [EP Rank 0/2] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 144/288. Experts local to global index map: 0->0, 1->1, 2->2, 3->3, 4->4, 5->5, 6->6, 7->7, 8->8, 9->9, 10->10, 11->11, 12->12, 13->13, 14->14, 15->15, 16->16, 17->17, 18->18, 19->19, 20->20, 21->21, 22->22, 23->23, 24->24, 25->25, 26->26, 27->27, 28->28, 29->29, 30->30, 31->31, 32->32, 33->33, 34->34, 35->35, 36->36, 37->37, 38->38, 39->39, 40->40, 41->41, 42->42, 43->43, 44->44, 45->45, 46->46, 47->47, 48->48, 49->49, 50->50, 51->51, 52->52, 53->53, 54->54, 55->55, 56->56, 57->57, 58->58, 59->59, 60->60, 61->61, 62->62, 63->63, 64->64, 65->65, 66->66, 67->67, 68->68, 69->69, 70->70, 71->71, 72->72, 73->73, 74->74, 75->75, 76->76, 77->77, 78->78, 79->79, 80->80, 81->81, 82->82, 83->83, 84->84, 85->85, 86->86, 87->87, 88->88, 89->89, 90->90, 91->91, 92->92, 93->93, 94->94, 95->95, 96->96, 97->97, 98->98, 99->99, 100->100, 101->101, 102->102, 103->103, 104->104, 105->105, 106->106, 107->107, 108->108, 109->109, 110->110, 111->111, 112->112, 113->113, 114->114, 115->115, 116->116, 117->117, 118->118, 119->119, 120->120, 121->121, 122->122, 123->123, 124->124, 125->125, 126->126, 127->127, 128->128, 129->129, 130->130, 131->131, 132->132, 133->133, 134->134, 135->135, 136->136, 137->137, 138->138, 139->139, 140->140, 141->141, 142->142, 143->143.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [weight_utils.py:935] Filesystem type for checkpoints: XFS. Checkpoint size: 163.58 GiB. Available RAM: 45.66 GiB.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:22:32 [weight_utils.py:965] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre) and the checkpoint size (163.58 GiB) exceeds 90% of available RAM (45.66 GiB).
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:09 [default_loader.py:497] Loading weights took 36.85 seconds
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:11 [model_runner.py:469] Model loading took 80.58 GiB and 39.214554 seconds
(Worker_TP1_EP1 pid=264) INFO 09-23 17:23:11 [model_runner.py:469] Model loading took 80.58 GiB and 39.218591 seconds
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:11 [topk_topp_sampler.py:62] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:11 [interface.py:635] Setting kv cache block size to 64 for DEEPSEEK_V32_INDEXER backend.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:23:11 [interface.py:635] Setting kv cache block size to 64 for DEEPSEEK_V32_INDEXER backend.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:11 [interface.py:931] Setting attention block size to 3328 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:11 [interface.py:955] Padding mamba page size by 0.57% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:23:11 [interface.py:931] Setting attention block size to 3328 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:23:11 [interface.py:955] Padding mamba page size by 0.57% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=241) WARNING 09-23 17:23:11 [torch_utils.py:251] OMP_NUM_THREADS=2 is set; leaving Torch threads at 2 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:12 [encoder_runner.py:108] Encoder cache will be initialized with a budget of 7921 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0_EP0 pid=263) WARNING 09-23 17:23:13 [common.py:241] The bundled vllm_flash_attn package does not provide layers.rotary; using the native PyTorch rotary implementation.
(Worker_TP0_EP0 pid=263) 2026-09-23 17:23:22  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:23:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:23:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_post_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:23:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_post_tilelang`
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:33 [exl3.py:4824] EXL3 full-expert EP runtime planned: Trellis m=1..32 block_m=8, prefill trellis block_m=128 capacity=2048 arena=669.5MiB scheduler_capacity=2048 chunk=128 topk=8
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:34 [model_runner.py:1003] Profiled V1 prompt-logprobs workspace with chunk size 1024
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:35 [kernel_warmup.py:115] Warming up ll_bf16 router GEMM kernels for shapes: ((4096, 288),).
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:35 [kernel_warmup.py:488] Deferring runtime-dependent kernel warmup until KV cache initialization.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:23:38 [indexer.py:891] DSA indexer decode path: use_flattening=False use_varlen=False (next_n=1, use_fp4_indexer_cache=False)
(EngineCore pid=241) INFO 09-23 17:24:12 [shm_broadcast.py:801] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:21 [breakable_cudagraph.py:293] Breakable CUDA graph enabled
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:21  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_fused_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_fused_tilelang`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:31  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:33  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_fused_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:34  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_fused_tilelang`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:34  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP0_EP0 pid=263) 2026-09-23 17:24:39  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:23:22  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:23:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:23:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_post_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:23:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_post_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:21  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:26  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_fused_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_fused_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:27  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:31  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:33  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_fused_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:34  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_fused_tilelang`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:34  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:129): TileLang begins to compile kernel `mhc_pre_big_fuse_with_norm_tilelang` with `out_idx=None`
(Worker_TP1_EP1 pid=264) 2026-09-23 17:24:39  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:137): TileLang completes to compile kernel `mhc_pre_big_fuse_with_norm_tilelang`
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:44 [model_runner.py:1246] Estimated MRV2 CUDA graph memory: 0.25 GiB total (0.11 GiB retained in the reusable pool)
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:44 [gpu_worker.py:661] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9700 is equivalent to --gpu-memory-utilization=0.9674 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9726. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:44 [model_runner.py:1246] Estimated MRV2 CUDA graph memory: 0.25 GiB total (0.11 GiB retained in the reusable pool)
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:44 [gpu_worker.py:646] Available KV cache memory: 6.41 GiB
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:44 [gpu_worker.py:661] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9700 is equivalent to --gpu-memory-utilization=0.9674 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9726. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=241) INFO 09-23 17:24:44 [kv_cache_utils.py:2580] GPU KV cache size: 825,268 tokens, Maximum concurrency for 327,680 tokens per request: 2.52x
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:44 [kernel_warmup.py:625] Using FlashInfer autotune cache file: /cache/jit/cu133-torch213-vllm174c789e09-b12x12c426322c-lmcachee045d729bc/flashinfer-autotune/31bab91f6e82675aae8970504d4f53ef2fbe8ea4fcc514054abed69eb1895c59/autotune_configs.json
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:48 [model_runner.py:1318] Graph capturing finished in 4 secs, took 0.06 GiB
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:48 [gpu_worker.py:813] CUDA graph pool memory: 0.06 GiB (actual), 0.25 GiB (estimated), difference: 0.18 GiB (284.8%).
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:48 [gpu_worker.py:876] Free memory on device (94.03/95.06 GiB) on startup. Desired GPU memory utilization is (0.97, 92.21 GiB). Actual usage is 84.89 GiB for consumed memory (weights + non-torch), 0.91 GiB for peak activation, and 0.06 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory-bytes=6654121288` (6.2 GiB) to fit into requested memory, or `--kv-cache-memory-bytes=8610739200` (8.02 GiB) to fully utilize gpu memory. Current kv cache memory in use is 6.41 GiB.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:48 [model_runner.py:1318] Graph capturing finished in 4 secs, took 0.06 GiB
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:48 [gpu_worker.py:813] CUDA graph pool memory: 0.06 GiB (actual), 0.25 GiB (estimated), difference: 0.18 GiB (284.8%).
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:48 [gpu_worker.py:876] Free memory on device (94.03/95.06 GiB) on startup. Desired GPU memory utilization is (0.97, 92.21 GiB). Actual usage is 84.89 GiB for consumed memory (weights + non-torch), 0.91 GiB for peak activation, and 0.06 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory-bytes=6654121288` (6.2 GiB) to fit into requested memory, or `--kv-cache-memory-bytes=8610739200` (8.02 GiB) to fully utilize gpu memory. Current kv cache memory in use is 6.41 GiB.
(Worker_TP0_EP0 pid=263) INFO 09-23 17:24:49 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(Worker_TP1_EP1 pid=264) INFO 09-23 17:24:49 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(Worker_TP0_EP0 pid=263) WARNING 09-23 17:24:50 [torch_utils.py:251] OMP_NUM_THREADS=2 is set; leaving Torch threads at 2 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
(EngineCore pid=241) INFO 09-23 17:24:50 [core.py:367] init engine (profile, create kv cache, warmup model) took 99.25 s
(EngineCore pid=241) INFO 09-23 17:24:53 [kernel.py:310] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(EngineCore pid=241) INFO 09-23 17:24:53 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1) INFO 09-23 17:24:53 [api_server.py:685] Supported tasks: ['generate']
(APIServer pid=1) INFO 09-23 17:24:53 [parser_manager.py:37] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 09-23 17:24:53 [hf.py:540] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) INFO 09-23 17:24:54 [base.py:235] Multi-modal warmup completed in 1.322s
(APIServer pid=1) INFO 09-23 17:24:55 [base.py:235] Readonly multi-modal warmup completed in 0.987s
(APIServer pid=1) WARNING 09-23 17:24:55 [model.py:1723] Default vLLM sampling parameters have been overridden by /root/.cache/huggingface: `{'temperature': 1.0, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 09-23 17:24:55 [api_server.py:689] Starting vLLM server on http://0.0.0.0:8002
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:51] Available routes are:
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /load, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /version, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /health, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /ping, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /ping, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /v1/completions/derender, Methods: POST
(APIServer pid=1) INFO 09-23 17:24:55 [launcher.py:60] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO:     100.64.0.10:58102 - "GET /health HTTP/1.1" 200 OK