{"command":"models recipes logs --container thinkingcap-5090-candidate --tail 240 --topology PRIVATE_OPERATOR_PATH/candidate-topology.toml --transport local","context":{"command":"models-recipes-logs","command_host":"Primary Node","command_runtime":"primary-native","controller_endpoint":null,"controller_endpoint_kind":null,"execution_host":"Primary Node","execution_runtime":"primary-docker","gpu_role":"media-compute","gpu_uuid":"GPU-RTX5090-SYNTHETIC","overlay":null,"resource":"thinkingcap-model-operator","resource_endpoint":null,"resource_endpoint_kind":null,"resource_host":"Primary Node","resource_runtime":"primary-docker","target":null,"topology":"Primary Node","transport":"local"},"data":"(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \u2588 \u2588 \u2588\u2584 \u2584\u2588\n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \u2584\u2584 \u2584\u2588 \u2588 \u2588 \u2588 \u2580\u2584\u2580 \u2588 version 0.29.0\n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \u2588\u2584\u2588\u2580 \u2588 \u2588 \u2588 \u2588 model /root/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ/snapshots/f8fe157f207a13f977bc3d620ce10a3e9ba5ab11\n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \u2580\u2580 \u2580\u2580\u2580\u2580\u2580 \u2580\u2580\u2580\u2580\u2580 \u2580 \u2580\n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:347] \n(APIServer pid=1) INFO 09-24 01:11:42 [api_utils.py:286] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'port': 30000, 'model': '/root/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ/snapshots/f8fe157f207a13f977bc3d620ce10a3e9ba5ab11', 'max_model_len': 16384, 'enforce_eager': True, 'served_model_name': ['thinkingcap-awq-5090'], 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.87, 'enable_prefix_caching': False, 'limit_mm_per_prompt': {'image': 2, 'video': 0}, 'mm_processor_kwargs': {'size': {'longest_edge': 1048576, 'shortest_edge': 65536}}, 'max_num_batched_tokens': 1024, 'max_num_seqs': 1}\n(APIServer pid=1) INFO 09-24 01:11:42 [model.py:684] Resolved architecture: Qwen3_5ForConditionalGeneration\n(APIServer pid=1) INFO 09-24 01:11:42 [model.py:2021] Using max model len 16384\n(APIServer pid=1) INFO 09-24 01:11:49 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=1024.\n(APIServer pid=1) WARNING 09-24 01:11:49 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none\n(APIServer pid=1) WARNING 09-24 01:11:49 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n(APIServer pid=1) INFO 09-24 01:11:49 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])\n(APIServer pid=1) INFO 09-24 01:11:49 [vllm.py:1585] Cudagraph is disabled under eager mode\n(APIServer pid=1) INFO 09-24 01:11:49 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant\n(EngineCore pid=115) INFO 09-24 01:12:06 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='/root/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ/snapshots/f8fe157f207a13f977bc3d620ce10a3e9ba5ab11', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ/snapshots/f8fe157f207a13f977bc3d620ce10a3e9ba5ab11', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=thinkingcap-awq-5090, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')\n(EngineCore pid=115) INFO 09-24 01:12:07 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_db76ed19dd8743c0b51d89b523a8e55c backend=nccl\n(EngineCore pid=115) INFO 09-24 01:12:07 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A\n(EngineCore pid=115) INFO 09-24 01:12:07 [gpu_worker.py:429] Using V2 Model Runner\n(EngineCore pid=115) INFO 09-24 01:12:09 [model_runner.py:382] Loading model from scratch...\n(EngineCore pid=115) INFO 09-24 01:12:16 [cuda.py:551] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention\n(EngineCore pid=115) INFO 09-24 01:12:16 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.\n(EngineCore pid=115) INFO 09-24 01:12:16 [__init__.py:726] Selected CutlassFP8ScaledMMLinearKernel for CompressedTensorsW8A8Fp8\n(EngineCore pid=115) INFO 09-24 01:12:16 [deep_gemm.py:186] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm\n(EngineCore pid=115) INFO 09-24 01:12:16 [deep_gemm.py:213] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.\n(EngineCore pid=115) INFO 09-24 01:12:16 [deep_gemm.py:131] DeepGEMM E8M0 enabled on current platform.\n(EngineCore pid=115) INFO 09-24 01:12:16 [qwen_gdn_linear_attn.py:167] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).\n(EngineCore pid=115) INFO 09-24 01:12:16 [qwen_gdn_linear_attn.py:519] GDN decode kernel: cuda\n(EngineCore pid=115) INFO 09-24 01:12:16 [__init__.py:1106] Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM\n(EngineCore pid=115) INFO 09-24 01:12:16 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].\n(EngineCore pid=115) INFO 09-24 01:12:16 [flash_attn.py:897] Using FlashAttention version 2\n(EngineCore pid=115) INFO 09-24 01:12:16 [weight_utils.py:863] Filesystem type for checkpoints: EXT4. Checkpoint size: 21.81 GiB. Available RAM: 15.13 GiB.\n(EngineCore pid=115) INFO 09-24 01:12:16 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (21.81 GiB) exceeds 90% of available RAM (15.13 GiB).\n(EngineCore pid=115) INFO 09-24 01:13:04 [default_loader.py:430] Loading weights took 47.61 seconds\n(EngineCore pid=115) INFO 09-24 01:13:07 [model_runner.py:404] Model loading took 21.34 GiB memory and 57.920669 seconds\n(EngineCore pid=115) INFO 09-24 01:13:07 [topk_topp_sampler.py:62] Using FlashInfer for top-p & top-k sampling.\n(EngineCore pid=115) INFO 09-24 01:13:07 [interface.py:918] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size.\n(EngineCore pid=115) INFO 09-24 01:13:07 [interface.py:942] Padding mamba page size by 0.13% to ensure that mamba page size and attention page size are exactly equal.\n(EngineCore pid=115) INFO 09-24 01:13:07 [utils.py:306] Using LBNHC KV cache layout.\n(EngineCore pid=115) INFO 09-24 01:13:14 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 1024 tokens, and profiled with 1 image items of the maximum feature size.\n(EngineCore pid=115) INFO 09-24 01:14:18 [gpu_worker.py:625] Available KV cache memory: 5.67 GiB\n(EngineCore pid=115) INFO 09-24 01:14:18 [kv_cache_utils.py:2032] GPU KV cache size: 80,554 tokens, Maximum concurrency for 16,384 tokens per request: 4.92x\n(EngineCore pid=115) INFO 09-24 01:14:18 [kernel_warmup.py:124] JIT kernel warmup starting.\n(EngineCore pid=115) INFO 09-24 01:14:18 [kernel_warmup.py:134] JIT kernel warmup finished in 0.00s.\n(EngineCore pid=115) INFO 09-24 01:14:18 [qwen_triton_warmup.py:270] Warming up Qwen Triton kernels for model_type=qwen3_5_text.\n(EngineCore pid=115) INFO 09-24 01:14:20 [kernel_warmup.py:309] Using FlashInfer autotune cache file: /root/.cache/vllm/flashinfer_autotune_cache/0.6.18/120f/d58b66eca627f08d1acfcd68c693d7115a031de255f48f897fcdd320d844d14c/autotune_configs.json\n(EngineCore pid=115) INFO 09-24 01:14:20 [kernel_warmup.py:266] Running FlashInfer autotune with 1024 tokens.\n(EngineCore pid=115) INFO 09-24 01:14:52 [gpu_worker.py:860] Free memory on device (30.2/31.82 GiB) on startup. Desired GPU memory utilization is (0.87, 27.68 GiB). Actual usage is 21.71 GiB for consumed memory (weights + non-torch), 0.3 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=5932725535` (5.53 GiB) to fit into requested memory, or `--kv-cache-memory=8641081344` (8.05 GiB) to fully utilize gpu memory. Current kv cache memory in use is 5.67 GiB.\n(EngineCore pid=115) INFO 09-24 01:15:49 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.\n(EngineCore pid=115) INFO 09-24 01:15:49 [torch_utils.py:277] Reducing Torch threads from 8 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.\n(EngineCore pid=115) INFO 09-24 01:15:49 [core.py:368] init engine (profile, create kv cache, warmup model) took 162.52 s\n(EngineCore pid=115) WARNING 09-24 01:15:49 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none\n(EngineCore pid=115) WARNING 09-24 01:15:49 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n(EngineCore pid=115) INFO 09-24 01:15:49 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])\n(EngineCore pid=115) INFO 09-24 01:15:49 [vllm.py:1585] Cudagraph is disabled under eager mode\n(EngineCore pid=115) INFO 09-24 01:15:49 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant\n(APIServer pid=1) INFO 09-24 01:15:50 [entry.py:135] Supported tasks: ['generate']\n(APIServer pid=1) INFO 09-24 01:15:50 [parser_manager.py:36] \"auto\" tool choice has been enabled.\n(APIServer pid=1) INFO 09-24 01:15:51 [hf.py:547] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.\n(APIServer pid=1) INFO 09-24 01:16:05 [base.py:244] Multi-modal warmup completed in 13.507s\n(APIServer pid=1) INFO 09-24 01:16:05 [base.py:244] Readonly multi-modal warmup completed in 0.015s\n(APIServer pid=1) INFO 09-24 01:16:05 [entry.py:139] Starting vLLM server on http://0.0.0.0:30000\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:61] Available routes are:\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /openapi.json, Methods: GET, HEAD\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /docs, Methods: GET, HEAD\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /docs/oauth2-redirect, Methods: GET, HEAD\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /redoc, Methods: GET, HEAD\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /load, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /version, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /health, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /metrics, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /tokenize, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /detokenize, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/models, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /ping, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /ping, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /invocations, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/chat/completions, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/chat/completions/batch, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/responses, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/responses/{response_id}, Methods: GET\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/responses/{response_id}/cancel, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/completions, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/messages, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/messages/count_tokens, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /generative_scoring, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /scale_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /is_scaling_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/chat/completions/render, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/messages/render, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/completions/render, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/chat/completions/derender, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /v1/completions/derender, Methods: POST\n(APIServer pid=1) INFO 09-24 01:16:05 [launcher.py:70] Route: /inference/v1/generate, Methods: POST\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:60176 - \"GET /health HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:39098 - \"GET /health HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:39106 - \"GET /v1/models HTTP/1.1\" 200 OK\n(EngineCore pid=115) WARNING 09-24 01:16:35 [jit_monitor.py:141] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49726 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:16:36 [loggers.py:310] Engine 000: Avg prompt throughput: 2.8 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49742 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49744 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49760 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49752 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:16:46 [loggers.py:310] Engine 000: Avg prompt throughput: 2195.4 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Waiting: 16 reqs, GPU KV cache usage: 9.4%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49762 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49772 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49788 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49800 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49808 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:16:56 [loggers.py:310] Engine 000: Avg prompt throughput: 2192.3 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Waiting: 11 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49818 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49826 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49846 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49840 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:17:06 [loggers.py:310] Engine 000: Avg prompt throughput: 2192.2 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 1 reqs, Waiting: 7 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49854 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49858 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49860 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49862 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:17:16 [loggers.py:310] Engine 000: Avg prompt throughput: 2740.4 tokens/s, Avg generation throughput: 11.6 tokens/s, Running: 1 reqs, Waiting: 3 reqs, GPU KV cache usage: 9.4%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49870 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49876 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49892 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:49902 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:53404 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO 09-24 01:17:26 [loggers.py:310] Engine 000: Avg prompt throughput: 1672.3 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:53406 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(APIServer pid=1) INFO: PRIVATE_CONTAINER_GATEWAY:53414 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK\n(EngineCore pid=115) WARNING 09-24 01:17:29 [jit_monitor.py:141] Triton kernel JIT compilation during inference: _bilinear_pos_embed_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.\n(APIServer pid=1) INFO 09-24 01:17:36 [loggers.py:310] Engine 000: Avg prompt throughput: 63.8 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%\n","error":null,"ok":true,"warnings":["/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py:24: DeprecationWarning: `vllm.entrypoints.openai.api_server` is deprecated and will likely beunsupported in a future version. Use the corresponding function from `vllm.entrypoints.launchers` instead."," warnings.warn(","/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py:51: DeprecationWarning: The `python -m vllm.entrypoints.openai.api_server` command is deprecated and may be removed in a future release. Please use `vllm server` instead."," warnings.warn(","(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend=\"torchvision\"` instead of `use_fast=True`, or `backend=\"pil\"` instead of `use_fast=False`.","(EngineCore pid=115) ","Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00