containment-passed
[TensorRT-LLM][INFO] Set logger level to INFO
[TensorRT-LLM][INFO] Set logger level to INFO
2026-09-22 07:04:49  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:130): TileLang begins to compile kernel `kernel` with `out_idx=None`
2026-09-22 07:04:51  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:138): TileLang completes to compile kernel `kernel`
2026-09-22 07:04:49  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:130): TileLang begins to compile kernel `kernel` with `out_idx=None`
2026-09-22 07:04:51  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:138): TileLang completes to compile kernel `kernel`
[2026-09-22 07:05:03] INFO:     PRIVATE_ADDRESS_REDACTED:53196 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:04] INFO:     127.0.0.1:<redacted-port> - "GET /model_info HTTP/1.1" 200 OK
[2026-09-22 07:05:05] INFO:     PRIVATE_ADDRESS_REDACTED:53208 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:07] INFO:     PRIVATE_ADDRESS_REDACTED:53222 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:10] INFO:     PRIVATE_ADDRESS_REDACTED:53230 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:12] INFO:     PRIVATE_ADDRESS_REDACTED:53242 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:14] INFO:     PRIVATE_ADDRESS_REDACTED:51286 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:15] INFO:     PRIVATE_ADDRESS_REDACTED:51294 - "GET /v1/models HTTP/1.1" 200 OK
[2026-09-22 07:05:16] INFO:     PRIVATE_ADDRESS_REDACTED:51306 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:18] INFO:     PRIVATE_ADDRESS_REDACTED:51308 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:20] INFO:     PRIVATE_ADDRESS_REDACTED:51316 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-09-22 07:05:20] INFO:     127.0.0.1:<redacted-port> - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-09-22 07:05:20] INFO:     127.0.0.1:<redacted-port> - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-09-22 07:05:23] INFO:     PRIVATE_ADDRESS_REDACTED:51326 - "GET /health HTTP/1.1" 200 OK
[2026-09-22 07:05:29] INFO:     PRIVATE_ADDRESS_REDACTED:51828 - "GET /health HTTP/1.1" 200 OK
[2026-09-22 07:05:45] INFO:     PRIVATE_ADDRESS_REDACTED:44772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-09-22 07:05:45] INFO:     PRIVATE_ADDRESS_REDACTED:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/sgl-workspace/sglang/python/sglang/launch_server.py:56: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
  Example: sglang serve --model-path <model> [options]
  warnings.warn(
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-22 07:03:25] Breakable CUDA graph is incompatible with multimodal model; disabling prefill CUDA graph.
[2026-09-22 07:03:25] Setting page size to 64 for compressed QSA (full//ratio compressed addressing).
[2026-09-22 07:03:25] Attention backend not specified. Use flashinfer backend by default.
[2026-09-22 07:03:25] Use flashinfer_cutlass as MoE runner backend on SM120 for modelopt_fp4 (trtllm-gen MoE kernels are SM100-only)
[2026-09-22 07:03:27] server_args=ServerArgs(model_path='PRIVATE_PATH_REDACTED', tokenizer_path='PRIVATE_PATH_REDACTED', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=262144, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization='modelopt_fp4', quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.9, max_running_requests=4, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=2048, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, retraction_policy='length', schedule_conservativeness=1.0, page_size=64, c128_page_size=16, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr=None, nnodes=1, node_rank=0, tp_size=2, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dwdp_size=1, dcp_comm_backend='ag_rs', dcp_replicate_q_proj=None, enable_prefill_cp=False, cp_strategy=None, enable_dsa_cache_layer_split=False, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_cp_decode_attn_tp=False, enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=817636133, mlx_enable_sampling=False, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='0.0.0.0', port='<redacted-port>', fastapi_root_path='', smg_grpc_mode=False, grpc_mode=False, grpc_port=None, sidecar=None, sidecar_args=None, skip_server_warmup=False, warmups=None, enable_http2=False, http2_max_concurrent_streams=200, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='qwen38-flash-next-nvfp4-sglang-tp2-c4-256k-v1', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='auto', default_chat_template_kwargs=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser='auto', tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=False, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, smg_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='<redacted-loopback-bind>:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='flashinfer_cutlass', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, flashinfer_autotune_skip_ops=None, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='full', max_bs=4, bs=[1, 2, 4], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None), prefill=PhaseConfig(backend='disabled', max_bs=2048, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None)), cuda_graph_backend_decode=None, cuda_graph_backend_prefill=None, cuda_graph_max_bs_decode=4, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, enable_scattered_sconv=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_dflash_block_size=None, speculative_dspark_block_size=None, speculative_dspark_sps_table_path=None, speculative_dspark_confidence_sts_path=None, speculative_dspark_align_verify_tokens_to_graph_tier=False, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_kv_cache_dtype=None, speculative_draft_window_size=None, speculative_moe_runner_backend='flashinfer_cutlass', speculative_moe_a2a_backend=None, speculative_draft_model_quantization='modelopt_fp4', _speculative_draft_quantization_explicitly_set=False, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='flashinfer_cutlass', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', fuseep_mode=2, deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, expert_balancedness_report_mode='off', deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_waterfill=False, ep_join_mode=None, ep_join_rank_offset=0, elastic_ep_initial_size=None, max_ep_size=None, elastic_ep_scale_timeout=600, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_max_states_per_path=-1, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='extra_buffer', uses_mamba_radix_cache=True, mamba_track_interval=64, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, ple_offload_embedding=False, ple_offload_backend='pinned', ple_offload_dir=None, linear_attn_verify_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_linear_replayssm_spec=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, mm_processor_worker_num=0, mm_io_worker_num=0, allowed_media_domains=[], media_url_max_file_size_mb=64, mm_preprocess_cache_size_mb=None, trust_mm_content_hashes=False, limit_mm_data_per_request=None, enable_mm_global_cache=False, image_processor_backend='auto', mm_global_cache_backend='mooncake', disable_fast_image_processor=False, mm_feature_transport='cpu', keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, enable_flexkv=False, flexkv_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, dllm_fdfo=True, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_retraction_backup=None, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_attempts=0, encoder_only=False, language_only=False, language_model_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, startup_weight_load_mode='serial', custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, return_hidden_states_mode=None, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, weight_cache_mode='off', weight_cache_socket=None, weight_cache_timeout=1800, forward_hooks=None, msprobe_dump_config=None)
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-22 07:03:27] Auto-detected --reasoning-parser as 'qwen3' from chat template
[2026-09-22 07:03:27] Auto-detected --tool-call-parser as 'qwen3_coder' from chat template
[2026-09-22 07:03:29] Ignore import error when loading sglang.srt.multimodal.processors.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-22 07:03:29] Ignore import error when loading sglang.srt.models.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[2026-09-22 07:03:29] Multimodal data loading enabled with 16 worker threads (auto).
[2026-09-22 07:03:31] Multimodal processor concurrency enabled with 2 isolated worker threads (auto).
[2026-09-22 07:03:31] Using default HuggingFace chat template with detected content format: openai
[2026-09-22 07:03:31] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen3_coder
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-22 07:03:33 TP1] Ignore import error when loading sglang.srt.multimodal.processors.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[2026-09-22 07:03:33 TP0] Ignore import error when loading sglang.srt.multimodal.processors.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[2026-09-22 07:03:33 TP1] Multimodal data loading enabled with 16 worker threads (auto).
[2026-09-22 07:03:33 TP0] Multimodal data loading enabled with 16 worker threads (auto).
[2026-09-22 07:03:33 TP1] Init torch distributed begin.
[2026-09-22 07:03:33 TP0] Init torch distributed begin.
[2026-09-22 07:03:34 TP0] sglang is using nccl==2.29.7
[2026-09-22 07:03:34 TP0] Setup Custom allreduce failed with invalid literal for int() with base 10: 'GPU-A-REDACTED'. To silence this warning, specify --disable-custom-all-reduce explicitly.
[2026-09-22 07:03:34 TP1] Setup Custom allreduce failed with invalid literal for int() with base 10: 'GPU-A-REDACTED'. To silence this warning, specify --disable-custom-all-reduce explicitly.
[2026-09-22 07:03:34 TP0] Init torch distributed ends. elapsed=0.95 s, mem usage=0.41 GB
[2026-09-22 07:03:34 TP1] Init torch distributed ends. elapsed=0.97 s, mem usage=0.41 GB
[2026-09-22 07:03:34 TP1] Ignore import error when loading sglang.srt.models.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[2026-09-22 07:03:34 TP0] Ignore import error when loading sglang.srt.models.sarashina2_vision: cannot import name 'MultimodalDataItem' from 'sglang.srt.managers.mm_utils' (/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py)
[2026-09-22 07:03:35 TP0] Load weight begin. avail mem=93.92 GB
[2026-09-22 07:03:35 TP1] Load weight begin. avail mem=93.92 GB
[2026-09-22 07:03:35 TP0] Using ModelOptModelLoader due to ModelOpt quantization config.
[2026-09-22 07:03:35 TP0] ModelOptModelLoader: Loading base model...
[2026-09-22 07:03:35 TP0] Model is already quantized, loading directly...
[2026-09-22 07:03:35 TP0] Detected nvfp4 checkpoint. Please note that the format is experimental and subject to change.
[2026-09-22 07:03:35 TP1] Using ModelOptModelLoader due to ModelOpt quantization config.
[2026-09-22 07:03:35 TP1] ModelOptModelLoader: Loading base model...
[2026-09-22 07:03:35 TP1] Model is already quantized, loading directly...
[2026-09-22 07:03:35 TP1] Detected nvfp4 checkpoint. Please note that the format is experimental and subject to change.
[2026-09-22 07:03:35 TP0] Multimodal attention backend not set. Use triton_attn.
[2026-09-22 07:03:35 TP1] Multimodal attention backend not set. Use triton_attn.
[2026-09-22 07:03:35 TP0] Using triton_attn as multimodal attention backend.
[2026-09-22 07:03:35 TP1] Using triton_attn as multimodal attention backend.
[2026-09-22 07:03:35 TP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_cutlass, quant_method=ModelOptNvFp4FusedMoEMethod).
[2026-09-22 07:03:35 TP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_cutlass, quant_method=ModelOptNvFp4FusedMoEMethod).
[2026-09-22 07:03:35 TP0] using attn output gate!
[2026-09-22 07:03:35 TP1] using attn output gate!

Multi-thread loading shards:   0% Completed | 0/206 [00:00<?, ?it/s]
Multi-thread loading shards:   0% Completed | 1/206 [00:00<02:34,  1.33it/s]
Multi-thread loading shards:   1% Completed | 2/206 [00:00<01:25,  2.39it/s]
Multi-thread loading shards:   1% Completed | 3/206 [00:01<01:04,  3.15it/s]
Multi-thread loading shards:   2% Completed | 4/206 [00:01<00:56,  3.60it/s]
Multi-thread loading shards:   2% Completed | 5/206 [00:01<00:47,  4.21it/s]
Multi-thread loading shards:   3% Completed | 6/206 [00:01<00:43,  4.56it/s]
Multi-thread loading shards:   3% Completed | 7/206 [00:01<00:42,  4.70it/s]
Multi-thread loading shards:   4% Completed | 8/206 [00:02<00:42,  4.68it/s]
Multi-thread loading shards:   4% Completed | 9/206 [00:02<00:39,  5.00it/s]
Multi-thread loading shards:   5% Completed | 10/206 [00:02<00:37,  5.19it/s]
Multi-thread loading shards:   5% Completed | 11/206 [00:02<00:37,  5.21it/s]
Multi-thread loading shards:   6% Completed | 12/206 [00:02<00:38,  5.02it/s]
Multi-thread loading shards:   6% Completed | 13/206 [00:03<00:36,  5.27it/s]
Multi-thread loading shards:   7% Completed | 14/206 [00:03<00:37,  5.17it/s]
Multi-thread loading shards:   7% Completed | 15/206 [00:03<00:39,  4.89it/s]
Multi-thread loading shards:   8% Completed | 16/206 [00:03<00:39,  4.76it/s]
Multi-thread loading shards:   8% Completed | 17/206 [00:03<00:37,  5.07it/s]
Multi-thread loading shards:   9% Completed | 18/206 [00:04<00:37,  5.05it/s]
Multi-thread loading shards:   9% Completed | 19/206 [00:04<00:38,  4.90it/s]
Multi-thread loading shards:  10% Completed | 20/206 [00:04<00:39,  4.77it/s]
Multi-thread loading shards:  10% Completed | 21/206 [00:04<00:37,  4.92it/s]
Multi-thread loading shards:  11% Completed | 22/206 [00:04<00:37,  4.93it/s]
Multi-thread loading shards:  11% Completed | 23/206 [00:05<00:54,  3.33it/s]
Multi-thread loading shards:  12% Completed | 24/206 [00:05<00:46,  3.93it/s]
Multi-thread loading shards:  12% Completed | 25/206 [00:05<00:38,  4.75it/s]
Multi-thread loading shards:  13% Completed | 26/206 [00:05<00:37,  4.81it/s]
Multi-thread loading shards:  13% Completed | 27/206 [00:06<00:37,  4.73it/s]
Multi-thread loading shards:  14% Completed | 28/206 [00:06<00:40,  4.44it/s]
Multi-thread loading shards:  14% Completed | 29/206 [00:06<00:37,  4.78it/s]
Multi-thread loading shards:  15% Completed | 30/206 [00:06<00:36,  4.85it/s]
Multi-thread loading shards:  15% Completed | 31/206 [00:06<00:36,  4.84it/s]
Multi-thread loading shards:  16% Completed | 32/206 [00:07<00:35,  4.86it/s]
Multi-thread loading shards:  16% Completed | 33/206 [00:07<00:32,  5.27it/s]
Multi-thread loading shards:  17% Completed | 34/206 [00:07<00:32,  5.33it/s]
Multi-thread loading shards:  17% Completed | 35/206 [00:07<00:33,  5.06it/s]
Multi-thread loading shards:  17% Completed | 36/206 [00:07<00:34,  4.91it/s]
Multi-thread loading shards:  18% Completed | 37/206 [00:08<00:32,  5.28it/s]
Multi-thread loading shards:  18% Completed | 38/206 [00:08<00:30,  5.51it/s]
Multi-thread loading shards:  19% Completed | 39/206 [00:08<00:30,  5.43it/s]
Multi-thread loading shards:  19% Completed | 40/206 [00:08<00:32,  5.16it/s]
Multi-thread loading shards:  20% Completed | 41/206 [00:08<00:29,  5.53it/s]
Multi-thread loading shards:  20% Completed | 42/206 [00:08<00:28,  5.70it/s]
Multi-thread loading shards:  21% Completed | 43/206 [00:09<00:29,  5.54it/s]
Multi-thread loading shards:  21% Completed | 44/206 [00:09<00:31,  5.21it/s]
Multi-thread loading shards:  22% Completed | 45/206 [00:09<00:29,  5.43it/s]
Multi-thread loading shards:  22% Completed | 46/206 [00:09<00:30,  5.22it/s]
Multi-thread loading shards:  23% Completed | 47/206 [00:09<00:30,  5.15it/s]
Multi-thread loading shards:  23% Completed | 48/206 [00:10<00:31,  4.94it/s]
Multi-thread loading shards:  24% Completed | 49/206 [00:10<00:30,  5.14it/s]
Multi-thread loading shards:  24% Completed | 50/206 [00:10<00:30,  5.14it/s]
Multi-thread loading shards:  25% Completed | 51/206 [00:10<00:30,  5.05it/s]
Multi-thread loading shards:  25% Completed | 52/206 [00:10<00:31,  4.93it/s]
Multi-thread loading shards:  26% Completed | 53/206 [00:11<00:29,  5.16it/s]
Multi-thread loading shards:  26% Completed | 54/206 [00:11<00:29,  5.15it/s]
Multi-thread loading shards:  27% Completed | 55/206 [00:11<00:30,  5.03it/s]
Multi-thread loading shards:  27% Completed | 56/206 [00:11<00:31,  4.70it/s]
Multi-thread loading shards:  28% Completed | 57/206 [00:11<00:30,  4.96it/s]
Multi-thread loading shards:  28% Completed | 58/206 [00:12<00:29,  5.01it/s]
Multi-thread loading shards:  29% Completed | 59/206 [00:12<00:29,  4.94it/s]
Multi-thread loading shards:  29% Completed | 60/206 [00:12<00:29,  4.87it/s]
Multi-thread loading shards:  30% Completed | 61/206 [00:12<00:29,  4.99it/s]
Multi-thread loading shards:  30% Completed | 62/206 [00:12<00:28,  5.05it/s]
Multi-thread loading shards:  31% Completed | 63/206 [00:13<00:28,  4.95it/s]
Multi-thread loading shards:  31% Completed | 64/206 [00:13<00:29,  4.78it/s]
Multi-thread loading shards:  32% Completed | 65/206 [00:13<00:28,  5.00it/s]
Multi-thread loading shards:  32% Completed | 66/206 [00:13<00:27,  5.06it/s]
Multi-thread loading shards:  33% Completed | 67/206 [00:13<00:28,  4.92it/s]
Multi-thread loading shards:  33% Completed | 68/206 [00:14<00:29,  4.73it/s]
Multi-thread loading shards:  33% Completed | 69/206 [00:14<00:28,  4.83it/s]
Multi-thread loading shards:  34% Completed | 70/206 [00:14<00:27,  4.89it/s]
Multi-thread loading shards:  34% Completed | 71/206 [00:14<00:26,  5.01it/s]
Multi-thread loading shards:  35% Completed | 72/206 [00:15<00:28,  4.64it/s]
Multi-thread loading shards:  35% Completed | 73/206 [00:15<00:27,  4.80it/s]
Multi-thread loading shards:  36% Completed | 74/206 [00:15<00:26,  4.91it/s]
Multi-thread loading shards:  36% Completed | 75/206 [00:15<00:26,  4.98it/s]
Multi-thread loading shards:  37% Completed | 76/206 [00:15<00:26,  4.93it/s]
Multi-thread loading shards:  37% Completed | 77/206 [00:15<00:24,  5.19it/s]
Multi-thread loading shards:  38% Completed | 78/206 [00:16<00:24,  5.12it/s]
Multi-thread loading shards:  38% Completed | 79/206 [00:16<00:25,  4.89it/s]
Multi-thread loading shards:  39% Completed | 80/206 [00:16<00:25,  4.93it/s]
Multi-thread loading shards:  39% Completed | 81/206 [00:16<00:23,  5.24it/s]
Multi-thread loading shards:  40% Completed | 82/206 [00:16<00:24,  5.05it/s]
Multi-thread loading shards:  40% Completed | 83/206 [00:17<00:36,  3.39it/s]
Multi-thread loading shards:  41% Completed | 84/206 [00:17<00:30,  3.99it/s]
Multi-thread loading shards:  41% Completed | 85/206 [00:17<00:25,  4.79it/s]
Multi-thread loading shards:  42% Completed | 86/206 [00:17<00:24,  4.94it/s]
Multi-thread loading shards:  42% Completed | 87/206 [00:18<00:25,  4.62it/s]
Multi-thread loading shards:  43% Completed | 88/206 [00:18<00:26,  4.42it/s]
Multi-thread loading shards:  43% Completed | 89/206 [00:18<00:26,  4.48it/s]
Multi-thread loading shards:  44% Completed | 90/206 [00:18<00:24,  4.68it/s]
Multi-thread loading shards:  44% Completed | 91/206 [00:19<00:24,  4.75it/s]
Multi-thread loading shards:  45% Completed | 92/206 [00:19<00:25,  4.54it/s]
Multi-thread loading shards:  45% Completed | 93/206 [00:19<00:24,  4.60it/s]
Multi-thread loading shards:  46% Completed | 94/206 [00:19<00:23,  4.68it/s]
Multi-thread loading shards:  46% Completed | 95/206 [00:19<00:23,  4.66it/s]
Multi-thread loading shards:  47% Completed | 96/206 [00:20<00:24,  4.54it/s]
Multi-thread loading shards:  47% Completed | 97/206 [00:20<00:22,  4.82it/s]
Multi-thread loading shards:  48% Completed | 98/206 [00:20<00:22,  4.76it/s]
Multi-thread loading shards:  48% Completed | 99/206 [00:20<00:23,  4.58it/s]
Multi-thread loading shards:  49% Completed | 100/206 [00:21<00:22,  4.64it/s]
Multi-thread loading shards:  49% Completed | 101/206 [00:21<00:21,  5.00it/s]
Multi-thread loading shards:  50% Completed | 102/206 [00:21<00:19,  5.23it/s]
Multi-thread loading shards:  50% Completed | 103/206 [00:21<00:20,  5.09it/s]
Multi-thread loading shards:  50% Completed | 104/206 [00:21<00:20,  5.00it/s]
Multi-thread loading shards:  51% Completed | 105/206 [00:21<00:19,  5.15it/s]
Multi-thread loading shards:  51% Completed | 106/206 [00:22<00:18,  5.31it/s]
Multi-thread loading shards:  52% Completed | 107/206 [00:22<00:19,  5.16it/s]
Multi-thread loading shards:  52% Completed | 108/206 [00:22<00:19,  5.12it/s]
Multi-thread loading shards:  53% Completed | 109/206 [00:22<00:17,  5.41it/s]
Multi-thread loading shards:  53% Completed | 110/206 [00:22<00:18,  5.31it/s]
Multi-thread loading shards:  54% Completed | 111/206 [00:23<00:18,  5.09it/s]
Multi-thread loading shards:  54% Completed | 112/206 [00:23<00:19,  4.87it/s]
Multi-thread loading shards:  55% Completed | 113/206 [00:23<00:18,  5.09it/s]
Multi-thread loading shards:  55% Completed | 114/206 [00:23<00:17,  5.31it/s]
Multi-thread loading shards:  56% Completed | 115/206 [00:23<00:18,  4.99it/s]
Multi-thread loading shards:  56% Completed | 116/206 [00:24<00:18,  4.77it/s]
Multi-thread loading shards:  57% Completed | 117/206 [00:24<00:18,  4.92it/s]
Multi-thread loading shards:  57% Completed | 118/206 [00:24<00:17,  5.12it/s]
Multi-thread loading shards:  58% Completed | 119/206 [00:24<00:17,  4.99it/s]
Multi-thread loading shards:  58% Completed | 120/206 [00:24<00:17,  4.95it/s]
Multi-thread loading shards:  59% Completed | 121/206 [00:25<00:16,  5.15it/s]
Multi-thread loading shards:  59% Completed | 122/206 [00:25<00:16,  5.13it/s]
Multi-thread loading shards:  60% Completed | 123/206 [00:25<00:16,  4.96it/s]
Multi-thread loading shards:  60% Completed | 124/206 [00:25<00:16,  4.91it/s]
Multi-thread loading shards:  61% Completed | 125/206 [00:25<00:15,  5.15it/s]
Multi-thread loading shards:  61% Completed | 126/206 [00:26<00:15,  5.13it/s]
Multi-thread loading shards:  62% Completed | 127/206 [00:26<00:15,  5.04it/s]
Multi-thread loading shards:  62% Completed | 128/206 [00:26<00:16,  4.72it/s]
Multi-thread loading shards:  63% Completed | 129/206 [00:26<00:15,  4.93it/s]
Multi-thread loading shards:  63% Completed | 130/206 [00:26<00:15,  4.87it/s]
Multi-thread loading shards:  64% Completed | 131/206 [00:27<00:15,  4.81it/s]
Multi-thread loading shards:  64% Completed | 132/206 [00:27<00:15,  4.73it/s]
Multi-thread loading shards:  65% Completed | 133/206 [00:27<00:14,  4.99it/s]
Multi-thread loading shards:  65% Completed | 134/206 [00:27<00:14,  5.07it/s]
Multi-thread loading shards:  66% Completed | 135/206 [00:27<00:14,  5.01it/s]
Multi-thread loading shards:  66% Completed | 136/206 [00:28<00:14,  4.77it/s]
Multi-thread loading shards:  67% Completed | 137/206 [00:28<00:14,  4.93it/s]
Multi-thread loading shards:  67% Completed | 138/206 [00:28<00:13,  4.92it/s]
Multi-thread loading shards:  67% Completed | 139/206 [00:28<00:14,  4.75it/s]
Multi-thread loading shards:  68% Completed | 140/206 [00:29<00:14,  4.57it/s]
Multi-thread loading shards:  68% Completed | 141/206 [00:29<00:13,  4.79it/s]
Multi-thread loading shards:  69% Completed | 142/206 [00:29<00:13,  4.80it/s]
Multi-thread loading shards:  69% Completed | 143/206 [00:29<00:19,  3.26it/s]
Multi-thread loading shards:  70% Completed | 144/206 [00:30<00:16,  3.83it/s]
Multi-thread loading shards:  70% Completed | 145/206 [00:30<00:13,  4.56it/s]
Multi-thread loading shards:  71% Completed | 146/206 [00:30<00:12,  4.64it/s]
Multi-thread loading shards:  71% Completed | 147/206 [00:30<00:13,  4.25it/s]
Multi-thread loading shards:  72% Completed | 148/206 [00:30<00:14,  4.01it/s]
Multi-thread loading shards:  72% Completed | 149/206 [00:31<00:13,  4.17it/s]
Multi-thread loading shards:  73% Completed | 150/206 [00:31<00:13,  4.28it/s]
Multi-thread loading shards:  73% Completed | 151/206 [00:31<00:12,  4.32it/s]
Multi-thread loading shards:  74% Completed | 152/206 [00:31<00:12,  4.35it/s]
Multi-thread loading shards:  74% Completed | 153/206 [00:32<00:11,  4.62it/s]
Multi-thread loading shards:  75% Completed | 154/206 [00:32<00:10,  4.73it/s]
Multi-thread loading shards:  75% Completed | 155/206 [00:32<00:10,  4.64it/s]
Multi-thread loading shards:  76% Completed | 156/206 [00:32<00:11,  4.52it/s]
Multi-thread loading shards:  76% Completed | 157/206 [00:32<00:10,  4.78it/s]
Multi-thread loading shards:  77% Completed | 158/206 [00:33<00:09,  4.84it/s]
Multi-thread loading shards:  77% Completed | 159/206 [00:33<00:09,  4.71it/s]
Multi-thread loading shards:  78% Completed | 160/206 [00:33<00:09,  4.63it/s]
Multi-thread loading shards:  78% Completed | 161/206 [00:33<00:09,  4.82it/s]
Multi-thread loading shards:  79% Completed | 162/206 [00:33<00:09,  4.76it/s]
Multi-thread loading shards:  79% Completed | 163/206 [00:34<00:09,  4.71it/s]
Multi-thread loading shards:  80% Completed | 164/206 [00:34<00:09,  4.58it/s]
Multi-thread loading shards:  80% Completed | 165/206 [00:34<00:08,  4.81it/s]
Multi-thread loading shards:  81% Completed | 166/206 [00:34<00:08,  4.87it/s]
Multi-thread loading shards:  81% Completed | 167/206 [00:35<00:08,  4.78it/s]
Multi-thread loading shards:  82% Completed | 168/206 [00:35<00:08,  4.66it/s]
Multi-thread loading shards:  82% Completed | 169/206 [00:35<00:07,  4.81it/s]
Multi-thread loading shards:  83% Completed | 170/206 [00:35<00:07,  4.75it/s]
Multi-thread loading shards:  83% Completed | 171/206 [00:35<00:07,  4.75it/s]
Multi-thread loading shards:  83% Completed | 172/206 [00:36<00:07,  4.59it/s]
Multi-thread loading shards:  84% Completed | 173/206 [00:36<00:06,  4.84it/s]
Multi-thread loading shards:  84% Completed | 174/206 [00:36<00:06,  4.86it/s]
Multi-thread loading shards:  85% Completed | 175/206 [00:36<00:06,  4.80it/s]
Multi-thread loading shards:  85% Completed | 176/206 [00:36<00:06,  4.43it/s]
Multi-thread loading shards:  86% Completed | 177/206 [00:37<00:06,  4.74it/s]
Multi-thread loading shards:  86% Completed | 178/206 [00:37<00:05,  4.84it/s]
Multi-thread loading shards:  87% Completed | 179/206 [00:37<00:05,  4.96it/s]
Multi-thread loading shards:  87% Completed | 180/206 [00:37<00:05,  4.77it/s]
Multi-thread loading shards:  88% Completed | 181/206 [00:37<00:05,  4.99it/s]
Multi-thread loading shards:  88% Completed | 182/206 [00:38<00:04,  4.90it/s]
Multi-thread loading shards:  89% Completed | 183/206 [00:38<00:04,  4.71it/s]
Multi-thread loading shards:  89% Completed | 184/206 [00:38<00:04,  5.06it/s]
Multi-thread loading shards:  90% Completed | 185/206 [00:38<00:03,  5.92it/s]
Multi-thread loading shards:  90% Completed | 186/206 [00:38<00:03,  5.80it/s]
Multi-thread loading shards:  91% Completed | 187/206 [00:38<00:03,  6.19it/s]
Multi-thread loading shards:  91% Completed | 188/206 [00:39<00:02,  6.44it/s]
Multi-thread loading shards:  92% Completed | 189/206 [00:39<00:02,  6.94it/s]
Multi-thread loading shards:  92% Completed | 190/206 [00:39<00:02,  7.39it/s]
Multi-thread loading shards:  93% Completed | 191/206 [00:39<00:02,  7.48it/s]
Multi-thread loading shards:  93% Completed | 192/206 [00:39<00:01,  7.37it/s]
Multi-thread loading shards:  94% Completed | 193/206 [00:39<00:02,  6.02it/s]
Multi-thread loading shards:  95% Completed | 195/206 [00:40<00:03,  3.17it/s]
Multi-thread loading shards:  95% Completed | 196/206 [00:41<00:03,  2.65it/s]
Multi-thread loading shards:  96% Completed | 197/206 [00:41<00:03,  2.68it/s]
Multi-thread loading shards:  96% Completed | 198/206 [00:42<00:02,  2.72it/s]
Multi-thread loading shards:  97% Completed | 199/206 [00:43<00:03,  1.84it/s]
Multi-thread loading shards:  97% Completed | 200/206 [00:44<00:05,  1.14it/s]
Multi-thread loading shards:  98% Completed | 201/206 [00:46<00:05,  1.00s/it]
Multi-thread loading shards:  98% Completed | 202/206 [00:47<00:04,  1.15s/it]
Multi-thread loading shards:  99% Completed | 203/206 [00:48<00:03,  1.16s/it]
Multi-thread loading shards:  99% Completed | 204/206 [00:48<00:01,  1.16it/s]
Multi-thread loading shards: 100% Completed | 206/206 [00:49<00:00,  2.02it/s]
Multi-thread loading shards: 100% Completed | 206/206 [00:49<00:00,  4.20it/s]
[2026-09-22 07:04:31 TP0] Load weight end. elapsed=56.77 s, type=Qwen4ExpForConditionalGeneration, quant=modelopt_fp4, quant_algo=NVFP4, avail mem=28.37 GB, mem usage=65.55 GB.
[2026-09-22 07:04:31 TP1] Load weight end. elapsed=56.79 s, type=Qwen4ExpForConditionalGeneration, quant=modelopt_fp4, quant_algo=NVFP4, avail mem=28.37 GB, mem usage=65.55 GB.
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
[2026-09-22 07:04:32 TP1] Reserving 0.10 GB of the KV budget for post-sizing multimodal allocations (feature-transport pools + embedding cache).
[2026-09-22 07:04:32 TP0] Reserving 0.10 GB of the KV budget for post-sizing multimodal allocations (feature-transport pools + embedding cache).
[2026-09-22 07:04:32 TP0] Mamba Cache is allocated. max_mamba_cache_size: 188, conv_state size: 0.19GB, ssm_state size: 9.97GB 
[2026-09-22 07:04:32 TP1] Mamba Cache is allocated. max_mamba_cache_size: 188, conv_state size: 0.19GB, ssm_state size: 9.97GB 
[2026-09-22 07:04:32 TP0] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 928576, K size: 5.31 GB, V size: 5.31 GB
[2026-09-22 07:04:32 TP1] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 928576, K size: 5.31 GB, V size: 5.31 GB
[2026-09-22 07:04:32 TP0] Memory pool end. avail mem=9.49 GB
[2026-09-22 07:04:32 TP1] Memory pool end. avail mem=9.49 GB
[2026-09-22 07:04:32 TP1] Using hybrid linear attention backend for hybrid GDN models.
[2026-09-22 07:04:32 TP0] Linear attention kernel backend: decode=triton, prefill=triton, verify=triton
[2026-09-22 07:04:32 TP0] Using hybrid linear attention backend for hybrid GDN models.
[2026-09-22 07:04:32 TP0] GDN kernel dispatcher: decode=TritonGDNKernel, extend=TritonGDNKernel, verify=TritonGDNKernel packed_decode=True
[2026-09-22 07:04:32 TP1] Using QSA for sparse full-attention layers.
[2026-09-22 07:04:32 TP0] Using QSA for sparse full-attention layers.
[2026-09-22 07:04:32 TP1] Running FlashInfer autotune with cache: PRIVATE_PATH_REDACTED
[2026-09-22 07:04:32 TP0] Running FlashInfer autotune with cache: PRIVATE_PATH_REDACTED

[AutoTuner]: Tuning trtllm::fused_moe::gemm1:   0%|          | 0/3 [00:00<?, ?profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1:   0%|          | 0/3 [00:00<?, ?profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1:  33%|███▎      | 1/3 [00:00<00:00,  9.35profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1:  33%|███▎      | 1/3 [00:00<00:00,  9.35profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 3/3 [00:00<00:00, 11.78profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 3/3 [00:00<00:00, 11.48profile/s]

[AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 3/3 [00:00<00:00, 11.77profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 3/3 [00:00<00:00, 11.47profile/s]

[AutoTuner]: Tuning trtllm::fused_moe::gemm2:   0%|          | 0/3 [00:00<?, ?profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2:   0%|          | 0/3 [00:00<?, ?profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2:  33%|███▎      | 1/3 [00:00<00:00,  5.72profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2:  33%|███▎      | 1/3 [00:00<00:00,  5.72profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2:  67%|██████▋   | 2/3 [00:00<00:00,  6.35profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2:  67%|██████▋   | 2/3 [00:00<00:00,  6.35profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 3/3 [00:00<00:00,  6.29profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 3/3 [00:00<00:00,  6.24profile/s]

[AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 3/3 [00:00<00:00,  6.29profile/s]
[AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 3/3 [00:00<00:00,  6.23profile/s]
[[0707::04:0449:49] ] : Warning: : Warning: Data race detected: `Logits(bx, position)`is written by multiple threads in loop (token,), Example:
  batch = 1
  bx = 0
  by = 0
  context_len = 1
  group = 0
  max_pages = 1
  position = 0
  token = 1
  token_OTHER_ = 0
  tx = 0
If you believe this is a false positive, pass `PassKey.TL_DISABLE_DATA_RACE_CHECK` to pass key to disable this check.Data race detected: `Logits(bx, position)`is written by multiple threads in loop (token,), Example:
  batch = 1
  bx = 0
  by = 0
  context_len = 1
  group = 0
  max_pages = 1
  position = 0
  token = 1
  token_OTHER_ = 0
  tx = 0
If you believe this is a false positive, pass `PassKey.TL_DISABLE_DATA_RACE_CHECK` to pass key to disable this check.

[2026-09-22 07:05:01 TP0] FlashInfer autotune completed.
[2026-09-22 07:05:01 TP1] FlashInfer autotune completed.
[2026-09-22 07:05:01 TP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
[2026-09-22 07:05:01 TP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
[2026-09-22 07:05:01 TP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4], avail mem=8.76 GB
[2026-09-22 07:05:01 TP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4], avail mem=8.76 GB

  0%|          | 0/3 [00:00<?, ?it/s]
Capturing batches (bs=4 avail_mem=8.76 GB):   0%|          | 0/3 [00:00<?, ?it/s][2026-09-22 07:05:01 TP1] multimem all-gather disabled (no multicast for world_size=2)
[2026-09-22 07:05:01 TP0] multimem all-gather disabled (no multicast for world_size=2)

Capturing batches (bs=4 avail_mem=8.76 GB):  33%|███▎      | 1/3 [00:00<00:00,  3.91it/s]
Capturing batches (bs=2 avail_mem=8.63 GB):  33%|███▎      | 1/3 [00:00<00:00,  3.91it/s]
Capturing batches (bs=2 avail_mem=8.63 GB):  67%|██████▋   | 2/3 [00:00<00:00,  5.18it/s]
Capturing batches (bs=1 avail_mem=8.62 GB):  67%|██████▋   | 2/3 [00:00<00:00,  5.18it/s]
Capturing batches (bs=1 avail_mem=8.62 GB): 100%|██████████| 3/3 [00:01<00:00,  1.77it/s]
Capturing batches (bs=1 avail_mem=8.62 GB): 100%|██████████| 3/3 [00:01<00:00,  2.12it/s]
[2026-09-22 07:05:03 TP1] Capture target decode CUDA graph end. elapsed=2.17 s, mem usage=0.14 GB, avail mem=8.62 GB.
[2026-09-22 07:05:03 TP1] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>, <ComponentType.MAMBA: 2>). Tree Core: UnifiedTreeCore
[2026-09-22 07:05:03 TP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=True hicache_attached=False streaming_wrapped=False
[2026-09-22 07:05:03 TP0] Capture target decode CUDA graph end. elapsed=2.17 s, mem usage=0.14 GB, avail mem=8.62 GB.
[2026-09-22 07:05:03 TP0] max_total_num_tokens=928576, chunked_prefill_size=2048, max_prefill_tokens=16384, max_running_requests=4, context_len=262144, available_gpu_mem=8.62 GB
[2026-09-22 07:05:03 TP0] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>, <ComponentType.MAMBA: 2>). Tree Core: UnifiedTreeCore
[2026-09-22 07:05:03 TP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=True hicache_attached=False streaming_wrapped=False
[2026-09-22 07:05:03] Engine startup timings (s): load_weight=56.79, kv_cache_allocation=0.42, scheduler_e2e=90.95, cuda_graph={prefill=0.00, decode=2.17, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=96.43
[2026-09-22 07:05:03] INFO:     Started server process [1]
[2026-09-22 07:05:03] INFO:     Waiting for application startup.
[2026-09-22 07:05:03] Using default chat sampling params from model generation config: {'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}
[2026-09-22 07:05:03] INFO:     Application startup complete.
[2026-09-22 07:05:03] INFO:     Uvicorn running on http://0.0.0.0:39132 (Press CTRL+C to quit)
/usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
/usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py:450: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  return await original_handler(ORJSONRequest(request.scope, request.receive))
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)
[2026-09-22 07:05:20 TP0] Prefill batch, #new-seq: 1, #new-token: 128, #cached-token: 0, full token usage: 0.00, mamba usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 7.32
[2026-09-22 07:05:20] Freezing GC in Tokenizer Manager process. gen0: 637->0, gen1: 2212->0, gen2: 1010084->0
[2026-09-22 07:05:20] The server is fired up and ready to roll!
[2026-09-22 07:05:20 TP0] Freezing GC in Scheduler process. gen0: 488->0, gen1: 312->0, gen2: 1206381->0
[2026-09-22 07:05:20 TP1] Freezing GC in Scheduler process. gen0: 488->0, gen1: 382->0, gen2: 1205748->0
[2026-09-22 07:05:20] Freezing GC in Detokenizer Manager process. gen0: 84->0, gen1: 0->0, gen2: 833834->0
/usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
/usr/local/lib/python3.12/dist-packages/triton/language/core.py:2284: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)
[2026-09-22 07:05:22 TP0] Prefill batch, #new-seq: 1, #new-token: 64, #cached-token: 0, full token usage: 0.00, mamba usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 35.49
[2026-09-22 07:05:28 TP0] Prefill batch, #new-seq: 1, #new-token: 64, #cached-token: 0, full token usage: 0.00, mamba usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 11.28
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)
[2026-09-22 07:05:43 TP0] Prefill batch, #new-seq: 1, #new-token: 128, #cached-token: 0, full token usage: 0.00, mamba usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 8.30
[2026-09-22 07:05:43 TP0] Decode batch, #running-req: 1, #full token: 128, full token usage: 0.00, mamba num: 4, mamba usage: 0.02, cuda graph: True, gen throughput (token/s): 0.98, #queue-req: 0
[2026-09-22 07:05:44 TP0] Decode batch, #running-req: 1, #full token: 192, full token usage: 0.00, mamba num: 4, mamba usage: 0.02, cuda graph: True, gen throughput (token/s): 107.77, #queue-req: 0
[2026-09-22 07:05:44 TP0] Decode batch, #running-req: 1, #full token: 192, full token usage: 0.00, mamba num: 4, mamba usage: 0.02, cuda graph: True, gen throughput (token/s): 107.36, #queue-req: 0
[2026-09-22 07:05:45 TP0] Decode batch, #running-req: 1, #full token: 256, full token usage: 0.00, mamba num: 4, mamba usage: 0.02, cuda graph: True, gen throughput (token/s): 106.97, #queue-req: 0
[2026-09-22 07:05:45 TP0] Prefill batch, #new-seq: 1, #new-token: 128, #cached-token: 0, full token usage: 0.00, mamba usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 83.20
[2026-09-22 07:05:45 TP0] Decode batch, #running-req: 1, #full token: 128, full token usage: 0.00, mamba num: 4, mamba usage: 0.02, cuda graph: True, gen throughput (token/s): 91.06, #queue-req: 0