containment-passed [TensorRT-LLM][INFO] Set logger level to INFO [TensorRT-LLM][INFO] Set logger level to INFO [2026-09-22 02:54:39] INFO: 127.0.0.1: - "GET /model_info HTTP/1.1" 200 OK [2026-09-22 02:54:40] INFO: PRIVATE_ADDRESS_REDACTED:42226 - "GET /health HTTP/1.1" 503 Service Unavailable [2026-09-22 02:54:42] INFO: PRIVATE_ADDRESS_REDACTED:42238 - "GET /health HTTP/1.1" 503 Service Unavailable [2026-09-22 02:54:44] INFO: PRIVATE_ADDRESS_REDACTED:35598 - "GET /health HTTP/1.1" 503 Service Unavailable [2026-09-22 02:54:45] INFO: 127.0.0.1: - "POST /v1/chat/completions HTTP/1.1" 200 OK [2026-09-22 02:54:45] INFO: 127.0.0.1: - "POST /freeze_gc HTTP/1.1" 200 OK [2026-09-22 02:54:47] INFO: PRIVATE_ADDRESS_REDACTED:35610 - "GET /health HTTP/1.1" 200 OK [2026-09-22 02:55:24] INFO: PRIVATE_ADDRESS_REDACTED:49228 - "GET /v1/models HTTP/1.1" 200 OK [2026-09-22 02:56:25] INFO: PRIVATE_ADDRESS_REDACTED:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2026-09-22 02:58:16] INFO: PRIVATE_ADDRESS_REDACTED:47070 - "GET /v1/models HTTP/1.1" 200 OK [2026-09-22 02:59:30] INFO: PRIVATE_ADDRESS_REDACTED:47082 - "POST /v1/chat/completions HTTP/1.1" 200 OK /sgl-workspace/sglang/python/sglang/launch_server.py:71: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint. Example: sglang serve --model-path [options] warnings.warn( /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( [transformers] The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details. [2026-09-22 02:52:47] Detected MiMo-V2 mxfp4 routed-expert layout. [2026-09-22 02:52:47] Hybrid SWA model detected. architectures=['MiMoV2ForCausalLM'] [2026-09-22 02:52:47] Breakable CUDA graph is incompatible with multimodal model; disabling prefill CUDA graph. [2026-09-22 02:52:47] Attention backend not specified. Use triton backend by default. [2026-09-22 02:52:49] server_args={'model_path': 'container-model-cache/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-RL/snapshots/3b38d063180c3e4aed9691fdc735f3d10b266ee4', 'tokenizer_path': 'container-model-cache/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-RL/snapshots/3b38d063180c3e4aed9691fdc735f3d10b266ee4', 'tokenizer_mode': 'auto', 'tokenizer_backend': 'huggingface', 'tokenizer_worker_num': 1, 'detokenizer_worker_num': 1, 'skip_tokenizer_init': False, 'load_format': 'auto', 'model_loader_extra_config': '{}', 'trust_remote_code': True, 'context_length': 327680, 'is_embedding': False, 'enable_multimodal': None, 'revision': None, 'model_impl': 'auto', 'model_config_parser': 'auto', 'json_model_override_args': '{}', 'dtype': 'auto', 'quantization': None, 'quantization_param_path': None, 'kv_cache_dtype': 'auto', 'enable_fp32_lm_head': False, 'modelopt_quant': None, 'modelopt_checkpoint_restore_path': None, 'modelopt_checkpoint_save_path': None, 'modelopt_export_path': None, 'quantize_and_serve': False, 'rl_quant_profile': None, 'enable_tf32_matmul': False, 'mem_fraction_static': 0.95, 'max_running_requests': 1, 'max_queued_requests': None, 'max_total_tokens': None, 'chunked_prefill_size': 2048, 'prefill_decode_interval': 0, 'enable_dynamic_chunking': False, 'max_prefill_tokens': 16384, 'prefill_max_requests': None, 'schedule_policy': 'fcfs', 'enable_priority_scheduling': False, 'disable_priority_preemption': False, 'default_priority_value': None, 'abort_on_priority_when_disabled': False, 'schedule_low_priority_values_first': False, 'priority_scheduling_preemption_threshold': 10, 'retraction_policy': 'length', 'schedule_conservativeness': 1.0, 'page_size': 1, 'c128_page_size': 16, 'swa_full_tokens_ratio': 0.8, 'disable_hybrid_swa_memory': False, 'radix_eviction_policy': 'lru', 'radix_eviction_policy_config': None, 'prefill_only_disable_kv_cache': False, 'disable_radix_cache': False, 'enable_page_major_kv_layout': False, 'enable_unified_memory': False, 'disable_chunked_prefix_cache': False, 'disable_overlap_schedule': False, 'num_continuous_decode_steps': 1, 'scheduler_recv_interval': 1, 'enable_mixed_chunk': False, 'nccl_port': None, 'dist_timeout': None, 'dist_init_addr': None, 'gated_launch_port': None, 'nnodes': 1, 'node_rank': 0, 'tp_size': 2, 'dcp_size': 1, 'pp_size': 1, 'pp_max_micro_batch_size': None, 'pp_async_batch_depth': 0, 'dp_size': 1, 'load_balance_method': 'round_robin', 'attn_cp_size': 1, 'moe_dp_size': 1, 'dwdp_size': 1, 'dcp_comm_backend': 'ag_rs', 'dcp_replicate_q_proj': None, 'enable_prefill_cp': False, 'cp_strategy': None, 'enable_dsa_cache_layer_split': False, 'enable_cp_decode_attn_tp': False, 'enable_dp_attention': False, 'enable_dp_attention_local_control_broadcast': False, 'enable_dp_lm_head': False, 'enable_tp_lm_head_all_to_all': False, 'enable_attn_tp_input_scattered': False, 'enable_shared_experts_attn_tp': False, 'enable_dense_mlp_attn_tp': False, 'enable_layernorm_sp': False, 'disable_attn_tp_gather': False, 'enable_p2p_check': False, 'device': 'cuda', 'base_gpu_id': 0, 'gpu_id_step': 1, 'random_seed': 136469088, 'mlx_enable_sampling': False, 'watchdog_timeout': 300, 'soft_watchdog_timeout': None, 'sleep_on_idle': False, 'use_ray': False, 'custom_sigquit_handler': None, 'numa_node': None, 'gc_threshold': None, 'host': '0.0.0.0', 'port: , 'fastapi_root_path': '', 'smg_grpc_mode': False, 'grpc_mode': False, 'grpc_port': None, 'grpc_worker_threads': 4, 'sidecar': None, 'sidecar_args': None, 'skip_server_warmup': False, 'warmups': None, 'enable_http2': False, 'http2_max_concurrent_streams': 200, 'http2_initial_connection_window_size': 1048576, 'ssl_keyfile': None, 'ssl_certfile': None, 'ssl_ca_certs': None, 'ssl_keyfile_password': None, 'enable_ssl_refresh': False, 'api_key': None, 'admin_api_key': None, 'served_model_name': 'mimo-v26-flash-rl-mxfp4-sglang-tp2-c1-327k', 'weight_version': 'default', 'chat_template': None, 'hf_chat_template_name': None, 'completion_template': None, 'file_storage_path': 'sglang_storage', 'enable_cache_report': False, 'return_input_ids': False, 'return_output_ids': False, 'reasoning_parser': 'mimo', 'default_chat_template_kwargs': None, 'strip_thinking_cache': False, 'enable_strict_thinking': False, 'tool_call_parser': 'mimo', 'tool_server': None, 'sampling_defaults': 'model', 'asr_max_buffer_seconds': 60, 'asr_max_concurrent_sessions': 32, 'preferred_sampling_params': None, 'allow_auto_truncate': False, 'stream_interval': 1, 'batch_notify_size': 16, 'stream_response_default_include_usage': False, 'incremental_streaming_output': False, 'enable_streaming_session': False, 'enable_session_radix_cache': False, 'log_level': 'info', 'log_level_http': None, 'log_requests': False, 'log_requests_level': 2, 'log_requests_format': 'text', 'log_requests_target': None, 'uvicorn_access_log_exclude_prefixes': [], 'crash_dump_folder': None, 'show_time_cost': False, 'enable_metrics': False, 'smg_http_sidecar_port': None, 'enable_mfu_metrics': False, 'enable_metrics_for_all_schedulers': False, 'load_snapshot_publish_interval': 15, 'tokenizer_metrics_custom_labels_header': 'x-custom-labels', 'tokenizer_metrics_allowed_custom_labels': None, 'extra_metric_labels': None, 'bucket_time_to_first_token': None, 'bucket_inter_token_latency': None, 'bucket_e2e_request_latency': None, 'prompt_tokens_buckets': None, 'generation_tokens_buckets': None, 'gc_warning_threshold_secs': 0.0, 'decode_log_interval': 40, 'enable_request_time_stats_logging': False, 'kv_events_config': None, 'load_publish_endpoint': None, 'enable_forward_pass_metrics': False, 'forward_pass_metrics_worker_id': '', 'forward_pass_metrics_ipc_name': None, 'enable_trace': False, 'trace_modules': 'request', 'otlp_traces_endpoint': '127.0.0.1:', 'export_metrics_to_file': False, 'export_metrics_to_file_dir': None, 'stat_loggers': None, 'constrained_json_whitespace_pattern': None, 'constrained_json_disable_any_whitespace': False, 'attention_backend': 'triton', 'decode_attention_backend': None, 'enable_lean_attention': None, 'prefill_attention_backend': None, 'sampling_backend': 'flashinfer', 'grammar_backend': 'xgrammar', 'radix_cache_backend': None, 'mm_attention_backend': None, 'fp8_gemm_runner_backend': 'auto', 'fp4_gemm_runner_backend': 'auto', 'bf16_gemm_backend': 'auto', 'dsa_prefill_backend': None, 'dsv4_prefill_backend': 'auto', 'dsa_decode_backend': None, 'dsa_paged_mqa_logits_backend': 'auto', 'dsa_topk_backend': 'sgl-kernel', 'disable_flashinfer_autotune': False, 'flashinfer_autotune_skip_ops': None, 'mamba_backend': 'triton', 'cuda_graph_config': {'decode': {'backend': 'full', 'max_bs': 1, 'bs': [1], 'tc_compiler': 'eager', 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None}, 'prefill': {'backend': 'disabled', 'max_bs': 2048, 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], 'tc_compiler': 'eager', 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None}}, 'cuda_graph_backend_decode': None, 'cuda_graph_backend_prefill': None, 'cuda_graph_max_bs_decode': 1, 'cuda_graph_max_bs_prefill': None, 'cuda_graph_bs_decode': None, 'cuda_graph_bs_prefill': None, 'cuda_graph_tc_compiler': None, 'disable_prefill_cuda_graph': False, 'disable_decode_cuda_graph': False, 'disable_cuda_graph': False, 'disable_cuda_graph_padding': False, 'enable_profile_cuda_graph': False, 'enable_cudagraph_gc': False, 'debug_cuda_graph': False, 'enable_layerwise_nvtx_marker': False, 'enable_nccl_nvls': False, 'enable_symm_mem': False, 'triton_attention_reduce_in_fp32': False, 'triton_attention_num_kv_splits': 8, 'triton_attention_split_tile_size': None, 'flashinfer_mla_disable_ragged': False, 'enable_fused_qk_norm_rope': False, 'enable_precise_embedding_interpolation': False, 'enable_fused_moe_sum_all_reduce': False, 'enable_deepseek_v4_fp4_indexer': False, 'disable_custom_all_reduce': False, 'enable_mscclpp': False, 'enable_torch_symm_mem': False, 'enable_scattered_sconv': False, 'pre_warm_nccl': False, 'enable_quant_communications': False, 'enable_flashinfer_allreduce_fusion': False, 'enforce_disable_flashinfer_allreduce_fusion': False, 'flashinfer_allreduce_fusion_backend': None, 'enable_aiter_allreduce_fusion': False, 'enable_torch_compile': False, 'enable_torch_compile_debug_mode': False, 'torch_compile_max_bs': 32, 'speculative_algorithm': None, 'uno_lora_path': None, 'speculative_draft_model_path': None, 'speculative_draft_model_revision': None, 'speculative_draft_load_format': None, 'speculative_num_steps': None, 'speculative_eagle_topk': None, 'speculative_num_draft_tokens': None, 'speculative_dflash_block_size': None, 'speculative_dspark_block_size': None, 'speculative_dspark_sps_table_path': None, 'speculative_dspark_confidence_sts_path': None, 'speculative_dspark_align_verify_tokens_to_graph_tier': False, 'speculative_accept_threshold_single': 1.0, 'speculative_accept_threshold_acc': 1.0, 'speculative_use_rejection_sampling': False, 'speculative_token_map': None, 'speculative_attention_mode': 'prefill', 'speculative_draft_attention_backend': None, 'speculative_dsa_topk_backend': 'sgl-kernel', 'speculative_draft_kv_cache_dtype': None, 'speculative_draft_window_size': None, 'speculative_moe_runner_backend': 'flashinfer_mxfp4', 'speculative_moe_a2a_backend': None, 'speculative_draft_model_quantization': None, '_speculative_draft_quantization_explicitly_set': False, 'speculative_skip_dp_mlp_sync': False, 'enable_multi_layer_eagle': False, 'speculative_adaptive': False, 'speculative_adaptive_config': None, 'decoupled_spec_bind_endpoint': None, 'decoupled_spec_connect_endpoints': None, 'decoupled_spec_rank': None, 'decoupled_spec_role': 'null', 'spec_trace_dir': None, 'speculative_ngram_min_bfs_breadth': 1, 'speculative_ngram_max_bfs_breadth': 10, 'speculative_ngram_match_type': 'BFS', 'speculative_ngram_max_trie_depth': 18, 'speculative_ngram_capacity': 10000000, 'speculative_ngram_external_corpus_path': None, 'speculative_ngram_external_sam_budget': 0, 'speculative_ngram_external_corpus_max_tokens': 10000000, 'ep_size': 1, 'moe_a2a_backend': 'none', 'enable_w4a4_mxfp4_megamoe': False, 'deepep_v2_mode': 'direct', 'moe_runner_backend': 'flashinfer_mxfp4', 'flashinfer_mxfp4_moe_precision': 'default', 'deepep_mode': 'auto', 'fuseep_mode': 2, 'deepep_dispatcher_output_dtype': 'auto', 'ep_num_redundant_experts': 0, 'ep_dispatch_algorithm': None, 'init_expert_location': 'trivial', 'enable_eplb': False, 'eplb_algorithm': 'auto', 'eplb_rebalance_num_iterations': 1000, 'eplb_rebalance_layers_per_chunk': None, 'eplb_min_rebalancing_utilization_threshold': 1.0, 'expert_distribution_recorder_mode': None, 'expert_distribution_recorder_buffer_size': 1000, 'expert_balancedness_report_mode': 'off', 'deepep_config': None, 'moe_dense_tp_size': None, 'elastic_ep_backend': None, 'enable_elastic_expert_backup': False, 'mooncake_ib_device': None, 'enable_waterfill': False, 'ep_join_mode': None, 'ep_join_rank_offset': 0, 'elastic_ep_initial_size': None, 'max_ep_size': None, 'elastic_ep_scale_timeout': 600, 'elastic_ep_rejoin': False, 'disable_flashinfer_cutlass_moe_fp4_allgather': False, 'disable_shared_experts_fusion': False, 'enforce_shared_experts_fusion': False, 'max_mamba_cache_size': None, 'mamba_ssm_dtype': None, 'mamba_max_states_per_path': -1, 'enable_mamba_cache_stochastic_rounding': False, 'mamba_cache_philox_rounds': 0, 'mamba_full_memory_ratio': 0.9, 'mamba_radix_cache_strategy': 'auto', 'uses_mamba_radix_cache': False, 'mamba_track_interval': 256, 'enable_int8_mamba_checkpoint': False, 'int8_mamba_ckpt_size': None, 'linear_attn_backend': 'triton', 'linear_attn_decode_backend': None, 'linear_attn_prefill_backend': None, 'linear_attn_verify_backend': None, 'enable_linear_replayssm': False, 'linear_replayssm_cache_len': 16, 'enable_linear_replayssm_spec': False, 'enable_hierarchical_cache': False, 'hicache_host_memory_mode': 'cache', 'hicache_ratio': 2.0, 'hicache_size': 0, 'hicache_write_policy': 'write_through', 'hicache_io_backend': 'kernel', 'hicache_mem_layout': 'page_first', 'hicache_storage_backend': None, 'hicache_storage_prefetch_policy': 'timeout', 'hicache_storage_backend_extra_config': None, 'hicache_storage_prefetch_retry_poll_interval': 8, 'hicache_storage_prefetch_retry_max_attempts': 8, 'enable_unified_cache_external_linker': False, 'unified_cache_external_linker_backend': 'mooncake', 'enable_hisparse': False, 'hisparse_config': None, 'enable_broadcast_mm_inputs_process': False, 'enable_prefix_mm_cache': False, 'mm_enable_dp_encoder': False, 'mm_process_config': {}, 'mm_processor_worker_num': 0, 'mm_io_worker_num': 0, 'allowed_media_domains': [], 'media_url_max_file_size_mb': 64, 'mm_preprocess_cache_size_mb': None, 'trust_mm_content_hashes': False, 'limit_mm_data_per_request': None, 'enable_mm_global_cache': False, 'image_processor_backend': 'auto', 'mm_global_cache_backend': 'mooncake', 'disable_fast_image_processor': False, 'mm_feature_transport': 'cpu', 'keep_mm_feature_on_device': False, 'enable_lora': None, 'enable_lora_overlap_loading': None, 'max_lora_rank': None, 'lora_target_modules': None, 'lora_paths': None, 'max_loaded_loras': None, 'max_loras_per_batch': 8, 'lora_eviction_policy': 'lru', 'lora_backend': 'csgmv', 'max_lora_chunk_size': 16, 'experts_shared_outer_loras': None, 'lora_use_virtual_experts': False, 'lora_strict_loading': False, 'lora_drain_wait_threshold': 0.0, 'enable_two_batch_overlap': False, 'enable_single_batch_overlap': False, 'tbo_token_distribution_threshold': 0.48, 'cpu_offload_gb': 0, 'offload_group_size': -1, 'offload_num_in_group': 1, 'offload_prefetch_step': 1, 'offload_mode': 'cpu', 'enable_lmcache': False, 'lmcache_config_file': None, 'enable_flexkv': False, 'flexkv_config_file': None, 'kt_weight_path': None, 'kt_method': 'AMXINT4', 'kt_cpuinfer': None, 'kt_threadpool_count': 2, 'kt_num_gpu_experts': None, 'kt_max_deferred_experts_per_token': None, 'dllm_algorithm': None, 'dllm_algorithm_config': None, 'dllm_fdfo': True, 'disaggregation_mode': 'null', 'disaggregation_transfer_backend': 'mooncake', 'disaggregation_bootstrap_port': 8998, 'disaggregation_ib_device': None, 'disaggregation_decode_enable_radix_cache': False, 'disaggregation_decode_enable_offload_kvcache': False, 'disaggregation_decode_retraction_backup': None, 'num_reserved_decode_tokens': 512, 'disaggregation_decode_extra_slots': None, 'disaggregation_decode_polling_interval': 1, 'optimistic_prefill_attempts': 0, 'encoder_only': False, 'language_only': False, 'language_model_only': False, 'encoder_transfer_backend': 'zmq_to_scheduler', 'encoder_urls': [], 'encoder_bootstrap_port': 8997, 'encoder_register_urls': [], 'enable_adaptive_dispatch_to_encoder': False, 'enable_pdmux': False, 'pdmux_config_path': None, 'sm_group_num': 8, 'startup_weight_load_mode': 'serial', 'custom_weight_loader': [], 'weight_loader_disable_mmap': False, 'weight_loader_prefetch_checkpoints': False, 'weight_loader_prefetch_num_threads': 4, 'weight_loader_drop_cache_after_load': False, 'remote_instance_weight_loader_seed_instance_ip': None, 'remote_instance_weight_loader_seed_instance_service_port': None, 'remote_instance_weight_loader_send_weights_group_ports': None, 'remote_instance_weight_loader_backend': 'nccl', 'remote_instance_weight_loader_start_seed_via_transfer_engine': False, 'engine_info_bootstrap_port': 6789, 'modelexpress_config': None, 'download_dir': None, 'model_checksum': None, 'delete_ckpt_after_loading': False, 'decrypted_config_file': None, 'decrypted_draft_config_file': None, 'checkpoint_engine_wait_weights_before_ready': False, 'enable_prefill_delayer': False, 'prefill_delayer_max_delay_passes': 30, 'prefill_delayer_token_usage_low_watermark': None, 'prefill_delayer_forward_passes_buckets': None, 'prefill_delayer_wait_seconds_buckets': None, 'prefill_delayer_queue_min_ratio': None, 'prefill_delayer_max_delay_ms': None, 'min_free_slots_delay': None, 'enable_deterministic_inference': False, 'rl_on_policy_target': None, 'kv_canary': 'none', 'kv_canary_real_data': 'none', 'kv_canary_sweep_interval': 0, 'enable_dynamic_batch_tokenizer': False, 'dynamic_batch_tokenizer_batch_size': 32, 'dynamic_batch_tokenizer_batch_timeout': 0.002, 'enable_tokenizer_batch_encode': False, 'disable_tokenizer_batch_decode': False, 'debug_tensor_dump_output_folder': None, 'debug_tensor_dump_layers': None, 'debug_tensor_dump_input_file': None, 'enable_memory_saver': False, 'enable_weights_cpu_backup': False, 'enable_draft_weights_cpu_backup': False, 'enable_custom_logit_processor': False, 'enable_return_hidden_states': False, 'return_hidden_states_mode': None, 'enable_return_routed_experts': False, 'enable_return_indexer_topk': False, 'sampling_mask_max_tokens': 4096, 'disable_outlines_disk_cache': False, 'enable_mis': False, 'weight_cache_mode': 'off', 'weight_cache_socket': None, 'weight_cache_timeout': 1800, 'forward_hooks': None, 'msprobe_dump_config': None, 'ple_offload_embedding': None, 'ple_offload_backend': 'pinned', 'ple_offload_dir': None, 'speculative_domino_candidate_pool_size': 2048, 'flashinfer_a2a_dispatch_type': None, 'dsv4_attn_backend': 'auto', '_radix_eviction_policy_explicitly_set': False, 'enable_response_store': False} [2026-09-22 02:52:49] Detected MiMo-V2 mxfp4 routed-expert layout. /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( [2026-09-22 02:52:52] Using default HuggingFace chat template with detected content format: openai [2026-09-22 02:52:52] Auto-detected template features: reasoning_parser=deepseek-r1, tool_call_parser=qwen3_coder /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( [transformers] The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details. [2026-09-22 02:52:54 TP1] Detected MiMo-V2 mxfp4 routed-expert layout. [transformers] The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details. [2026-09-22 02:52:54 TP0] Detected MiMo-V2 mxfp4 routed-expert layout. [2026-09-22 02:52:55 TP1] Detected MiMo-V2 mxfp4 routed-expert layout. [2026-09-22 02:52:55 TP0] Detected MiMo-V2 mxfp4 routed-expert layout. [2026-09-22 02:52:56 TP1] Init torch distributed begin. [2026-09-22 02:52:56 TP0] Init torch distributed begin. [2026-09-22 02:52:56 TP0] sglang is using nccl==2.30.7 [2026-09-22 02:52:57 TP0] Setup Custom allreduce failed with invalid literal for int() with base 10: 'GPU-REDACTED'. To silence this warning, specify --disable-custom-all-reduce explicitly. [2026-09-22 02:52:57 TP1] Setup Custom allreduce failed with invalid literal for int() with base 10: 'GPU-REDACTED'. To silence this warning, specify --disable-custom-all-reduce explicitly. [2026-09-22 02:52:57 TP1] Init torch distributed ends. elapsed=0.94 s, mem usage=0.41 GB [2026-09-22 02:52:57 TP0] Init torch distributed ends. elapsed=0.92 s, mem usage=0.41 GB [2026-09-22 02:52:57 TP1] Load weight begin. avail mem=93.99 GB [2026-09-22 02:52:57 TP0] Load weight begin. avail mem=93.99 GB [2026-09-22 02:52:57 TP0] Detected fp8 checkpoint. [2026-09-22 02:52:57 TP0] Skipping block quantization checks for weight partition. [2026-09-22 02:52:57 TP1] Skipping block quantization checks for weight partition. [2026-09-22 02:52:57 TP1] Multimodal attention backend not set. Use triton_attn. [2026-09-22 02:52:57 TP1] Using triton_attn as multimodal attention backend. [2026-09-22 02:52:57 TP0] Multimodal attention backend not set. Use triton_attn. [2026-09-22 02:52:57 TP0] Using triton_attn as multimodal attention backend. Multi-thread loading shards: 0% Completed | 0/65 [00:00 requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.0.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.1.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.1.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.10.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.10.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.11.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.11.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.12.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.12.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.13.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.13.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.14.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.14.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.15.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.15.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.16.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.16.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.17.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.17.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.18.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.18.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.19.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.19.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.2.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.2.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.20.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.20.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.21.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.21.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.22.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.22.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.23.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.23.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.24.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.24.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.25.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.25.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.26.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.26.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.27.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.27.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.28.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.28.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.29.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.29.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.3.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.3.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.30.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.30.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.31.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.31.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.32.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.32.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.33.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.33.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.34.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.34.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.35.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.35.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.36.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.36.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.37.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.37.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.38.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.38.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.39.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.39.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.4.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.4.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.40.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.40.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.41.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.41.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.42.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.42.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.43.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.43.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.44.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.44.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.45.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.45.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.46.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.46.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.47.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.47.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.5.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.5.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (53, 32), de-interleaved QKV (3072, 192, 128) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.6.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.6.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.7.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.7.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.8.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.8.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP1] Resolved deferred qkv scale_inv model.layers.9.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Resolved deferred qkv scale_inv model.layers.9.self_attn.qkv_proj.weight_scale_inv: dequant 2 shards -> requant (58, 32), de-interleaved QKV (3072, 384, 256) [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.1.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.2.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.3.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.4.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.5.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.6.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.7.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.8.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.9.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.10.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.11.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.12.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.13.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.14.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.15.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.16.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.17.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.18.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.19.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.20.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.21.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.22.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.23.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.24.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.25.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.26.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.27.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.28.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.29.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.30.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.31.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.32.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.33.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.34.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.35.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.36.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.37.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.38.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.39.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.40.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.41.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.42.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.43.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.44.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.45.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.46.mlp.experts)... [2026-09-22 02:53:39 TP0] Preparing DSv4 MXFP4 experts for FlashInfer SM120 CUTLASS W4A8 (layer: model.layers.47.mlp.experts)... [2026-09-22 02:53:39 TP0] Load weight end. elapsed=42.11 s, type=MiMoV2ForCausalLM, quant=fp8, fmt=e4m3, avail mem=12.58 GB, mem usage=81.41 GB. [2026-09-22 02:53:39 TP1] Load weight end. elapsed=42.11 s, type=MiMoV2ForCausalLM, quant=fp8, fmt=e4m3, avail mem=12.58 GB, mem usage=81.41 GB. /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( /opt/sglang/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime. warnings.warn( [2026-09-22 02:53:39 TP1] Reserving 0.10 GB of the KV budget for post-sizing multimodal allocations (feature-transport pools + embedding cache). [2026-09-22 02:53:39 TP0] Reserving 0.10 GB of the KV budget for post-sizing multimodal allocations (feature-transport pools + embedding cache). [2026-09-22 02:53:39 TP1] Use sliding window memory pool. full_layer_tokens=91342, swa_layer_tokens=73073 [2026-09-22 02:53:39 TP0] Use sliding window memory pool. full_layer_tokens=91342, swa_layer_tokens=73073 [2026-09-22 02:53:39 TP1] Full KV Cache is allocated. dtype: torch.bfloat16, #tokens: 91342, K size: 0.59 GB, V size: 0.39 GB [2026-09-22 02:53:39 TP0] Full KV Cache is allocated. dtype: torch.bfloat16, #tokens: 91342, K size: 0.59 GB, V size: 0.39 GB [2026-09-22 02:53:39 TP0] SWA KV Cache is allocated. dtype: torch.bfloat16, #tokens: 73073, K size: 4.08 GB, V size: 2.72 GB [2026-09-22 02:53:39 TP1] SWA KV Cache is allocated. dtype: torch.bfloat16, #tokens: 73073, K size: 4.08 GB, V size: 2.72 GB [2026-09-22 02:53:39 TP0] SWAKVPool mem usage: 7.77 GB, swa size: 73073, full size: 91342 [2026-09-22 02:53:39 TP1] SWAKVPool mem usage: 7.77 GB, swa size: 73073, full size: 91342 [2026-09-22 02:53:39 TP0] Memory pool end. avail mem=4.71 GB [2026-09-22 02:53:39 TP1] Memory pool end. avail mem=4.71 GB [2026-09-22 02:53:40 TP0] Running FlashInfer autotune with cache: container-runtime-cache/flashinfer/autotune/0.6.18/sm120/59c56bb05fbb6cf9/rank_tp0_pp0_dp0.json [2026-09-22 02:53:40 TP1] Running FlashInfer autotune with cache: container-runtime-cache/flashinfer/autotune/0.6.18/sm120/59c56bb05fbb6cf9/rank_tp1_pp0_dp0.json [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 0%| | 0/1 [00:00, ). Tree Core: UnifiedTreeCore [2026-09-22 02:54:38 TP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hicache_attached=False streaming_wrapped=False [2026-09-22 02:54:38 TP1] Init Unified Radix Cache. Components: (, ). Tree Core: UnifiedTreeCore [2026-09-22 02:54:38 TP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hicache_attached=False streaming_wrapped=False [2026-09-22 02:54:38] Engine startup timings (s): load_weight=42.11, kv_cache_allocation=0.37, scheduler_e2e=103.85, cuda_graph={prefill=0.00, decode=0.82, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=109.23 [2026-09-22 02:54:38] INFO: Started server process [1] [2026-09-22 02:54:38] INFO: Waiting for application startup. [2026-09-22 02:54:38] Using default chat sampling params from model generation config: {'temperature': 1.0, 'top_p': 0.95} [2026-09-22 02:54:38] INFO: Application startup complete. [2026-09-22 02:54:38] INFO: Uvicorn running on http://0.0.0.0: (Press CTRL+C to quit) /opt/sglang/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead. return func(*args, **kwargs) /opt/sglang/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead. return func(*args, **kwargs) [2026-09-22 02:54:45 TP0] Prefill batch, #new-seq: 1, #new-token: 22, #cached-token: 0, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 3.14 [2026-09-22 02:54:45] The server is fired up and ready to roll! /opt/sglang/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead. return func(*args, **kwargs) [2026-09-22 02:54:46 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1.09 [2026-09-22 02:55:10 TP0] Prefill batch, #new-seq: 1, #new-token: 20, #cached-token: 3, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.82 [2026-09-22 02:55:11 TP0] Decode batch, #running-req: 1, #full token: 55, full token usage: 0.00, #swa token: 55, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 1.23, #queue-req: 0 [2026-09-22 02:55:11 TP0] Decode batch, #running-req: 1, #full token: 95, full token usage: 0.00, #swa token: 95, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.93, #queue-req: 0 [2026-09-22 02:55:11 TP0] Decode batch, #running-req: 1, #full token: 135, full token usage: 0.00, #swa token: 135, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.16, #queue-req: 0 [2026-09-22 02:55:12 TP0] Decode batch, #running-req: 1, #full token: 175, full token usage: 0.00, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.22, #queue-req: 0 [2026-09-22 02:55:12 TP0] Decode batch, #running-req: 1, #full token: 215, full token usage: 0.00, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.27, #queue-req: 0 [2026-09-22 02:55:12 TP0] Decode batch, #running-req: 1, #full token: 255, full token usage: 0.00, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.94, #queue-req: 0 [2026-09-22 02:55:13 TP0] Decode batch, #running-req: 1, #full token: 295, full token usage: 0.00, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.57, #queue-req: 0 [2026-09-22 02:55:13 TP0] Decode batch, #running-req: 1, #full token: 335, full token usage: 0.00, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.84, #queue-req: 0 [2026-09-22 02:55:13 TP0] Decode batch, #running-req: 1, #full token: 375, full token usage: 0.00, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.85, #queue-req: 0 [2026-09-22 02:55:13 TP0] Decode batch, #running-req: 1, #full token: 415, full token usage: 0.00, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.61, #queue-req: 0 [2026-09-22 02:55:14 TP0] Decode batch, #running-req: 1, #full token: 455, full token usage: 0.00, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.56, #queue-req: 0 [2026-09-22 02:55:14 TP0] Decode batch, #running-req: 1, #full token: 495, full token usage: 0.01, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.50, #queue-req: 0 [2026-09-22 02:55:14 TP0] Decode batch, #running-req: 1, #full token: 535, full token usage: 0.01, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.56, #queue-req: 0 [2026-09-22 02:55:15 TP0] Decode batch, #running-req: 1, #full token: 575, full token usage: 0.01, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.58, #queue-req: 0 [2026-09-22 02:55:15 TP0] Decode batch, #running-req: 1, #full token: 615, full token usage: 0.01, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.59, #queue-req: 0 [2026-09-22 02:55:15 TP0] Decode batch, #running-req: 1, #full token: 655, full token usage: 0.01, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.42, #queue-req: 0 [2026-09-22 02:55:16 TP0] Decode batch, #running-req: 1, #full token: 695, full token usage: 0.01, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.54, #queue-req: 0 [2026-09-22 02:55:16 TP0] Decode batch, #running-req: 1, #full token: 735, full token usage: 0.01, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.59, #queue-req: 0 [2026-09-22 02:55:16 TP0] Decode batch, #running-req: 1, #full token: 775, full token usage: 0.01, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.44, #queue-req: 0 [2026-09-22 02:55:17 TP0] Decode batch, #running-req: 1, #full token: 815, full token usage: 0.01, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.29, #queue-req: 0 [2026-09-22 02:55:17 TP0] Decode batch, #running-req: 1, #full token: 855, full token usage: 0.01, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.43, #queue-req: 0 [2026-09-22 02:55:17 TP0] Decode batch, #running-req: 1, #full token: 895, full token usage: 0.01, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.35, #queue-req: 0 [2026-09-22 02:55:18 TP0] Decode batch, #running-req: 1, #full token: 935, full token usage: 0.01, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.88, #queue-req: 0 [2026-09-22 02:55:18 TP0] Decode batch, #running-req: 1, #full token: 975, full token usage: 0.01, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.92, #queue-req: 0 [2026-09-22 02:55:18 TP0] Decode batch, #running-req: 1, #full token: 1015, full token usage: 0.01, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.34, #queue-req: 0 [2026-09-22 02:55:18 TP0] Decode batch, #running-req: 1, #full token: 1055, full token usage: 0.01, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.20, #queue-req: 0 [2026-09-22 02:55:19 TP0] Decode batch, #running-req: 1, #full token: 1095, full token usage: 0.01, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.30, #queue-req: 0 [2026-09-22 02:55:19 TP0] Decode batch, #running-req: 1, #full token: 1135, full token usage: 0.01, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.66, #queue-req: 0 [2026-09-22 02:55:19 TP0] Decode batch, #running-req: 1, #full token: 1175, full token usage: 0.01, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.47, #queue-req: 0 [2026-09-22 02:55:20 TP0] Decode batch, #running-req: 1, #full token: 1215, full token usage: 0.01, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.17, #queue-req: 0 [2026-09-22 02:55:20 TP0] Decode batch, #running-req: 1, #full token: 1255, full token usage: 0.01, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.19, #queue-req: 0 [2026-09-22 02:55:20 TP0] Decode batch, #running-req: 1, #full token: 1295, full token usage: 0.01, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.04, #queue-req: 0 [2026-09-22 02:55:21 TP0] Decode batch, #running-req: 1, #full token: 1335, full token usage: 0.01, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.49, #queue-req: 0 [2026-09-22 02:55:21 TP0] Decode batch, #running-req: 1, #full token: 1375, full token usage: 0.02, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.39, #queue-req: 0 [2026-09-22 02:55:21 TP0] Decode batch, #running-req: 1, #full token: 1415, full token usage: 0.02, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.97, #queue-req: 0 [2026-09-22 02:55:22 TP0] Decode batch, #running-req: 1, #full token: 1455, full token usage: 0.02, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.88, #queue-req: 0 [2026-09-22 02:55:22 TP0] Decode batch, #running-req: 1, #full token: 1495, full token usage: 0.02, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.20, #queue-req: 0 [2026-09-22 02:55:22 TP0] Decode batch, #running-req: 1, #full token: 1535, full token usage: 0.02, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.25, #queue-req: 0 [2026-09-22 02:55:23 TP0] Decode batch, #running-req: 1, #full token: 1575, full token usage: 0.02, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.02, #queue-req: 0 [2026-09-22 02:55:23 TP0] Decode batch, #running-req: 1, #full token: 1615, full token usage: 0.02, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.10, #queue-req: 0 [2026-09-22 02:55:23 TP0] Decode batch, #running-req: 1, #full token: 1655, full token usage: 0.02, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.29, #queue-req: 0 [2026-09-22 02:55:23 TP0] Decode batch, #running-req: 1, #full token: 1695, full token usage: 0.02, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.21, #queue-req: 0 [2026-09-22 02:55:24 TP0] Decode batch, #running-req: 1, #full token: 1735, full token usage: 0.02, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.17, #queue-req: 0 /sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py:454: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/ return await original_handler(ORJSONRequest(request.scope, request.receive)) [2026-09-22 02:55:24 TP0] Decode batch, #running-req: 1, #full token: 1775, full token usage: 0.02, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.20, #queue-req: 0 [2026-09-22 02:55:24 TP0] Decode batch, #running-req: 1, #full token: 1815, full token usage: 0.02, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.22, #queue-req: 0 [2026-09-22 02:55:25 TP0] Decode batch, #running-req: 1, #full token: 1855, full token usage: 0.02, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.62, #queue-req: 0 [2026-09-22 02:55:25 TP0] Decode batch, #running-req: 1, #full token: 1895, full token usage: 0.02, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.45, #queue-req: 0 [2026-09-22 02:55:25 TP0] Decode batch, #running-req: 1, #full token: 1935, full token usage: 0.02, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.06, #queue-req: 0 [2026-09-22 02:55:26 TP0] Decode batch, #running-req: 1, #full token: 1975, full token usage: 0.02, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.05, #queue-req: 0 [2026-09-22 02:55:26 TP0] Decode batch, #running-req: 1, #full token: 2015, full token usage: 0.02, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.14, #queue-req: 0 [2026-09-22 02:55:26 TP0] Decode batch, #running-req: 1, #full token: 2055, full token usage: 0.02, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.06, #queue-req: 0 [2026-09-22 02:55:27 TP0] Decode batch, #running-req: 1, #full token: 2095, full token usage: 0.02, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.34, #queue-req: 0 [2026-09-22 02:55:27 TP0] Decode batch, #running-req: 1, #full token: 2135, full token usage: 0.02, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.36, #queue-req: 0 [2026-09-22 02:55:27 TP0] Decode batch, #running-req: 1, #full token: 2175, full token usage: 0.02, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.88, #queue-req: 0 [2026-09-22 02:55:28 TP0] Decode batch, #running-req: 1, #full token: 2215, full token usage: 0.02, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.91, #queue-req: 0 [2026-09-22 02:55:28 TP0] Decode batch, #running-req: 1, #full token: 2255, full token usage: 0.02, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.81, #queue-req: 0 [2026-09-22 02:55:28 TP0] Decode batch, #running-req: 1, #full token: 2295, full token usage: 0.03, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.79, #queue-req: 0 [2026-09-22 02:55:28 TP0] Decode batch, #running-req: 1, #full token: 2335, full token usage: 0.03, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.52, #queue-req: 0 [2026-09-22 02:55:29 TP0] Decode batch, #running-req: 1, #full token: 2375, full token usage: 0.03, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.65, #queue-req: 0 [2026-09-22 02:55:29 TP0] Decode batch, #running-req: 1, #full token: 2415, full token usage: 0.03, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.44, #queue-req: 0 [2026-09-22 02:55:29 TP0] Decode batch, #running-req: 1, #full token: 2455, full token usage: 0.03, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.44, #queue-req: 0 [2026-09-22 02:55:30 TP0] Decode batch, #running-req: 1, #full token: 2495, full token usage: 0.03, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.48, #queue-req: 0 [2026-09-22 02:55:30 TP0] Decode batch, #running-req: 1, #full token: 2535, full token usage: 0.03, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.50, #queue-req: 0 [2026-09-22 02:55:30 TP0] Decode batch, #running-req: 1, #full token: 2575, full token usage: 0.03, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.53, #queue-req: 0 [2026-09-22 02:55:31 TP0] Decode batch, #running-req: 1, #full token: 2615, full token usage: 0.03, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.67, #queue-req: 0 [2026-09-22 02:55:31 TP0] Decode batch, #running-req: 1, #full token: 2655, full token usage: 0.03, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.57, #queue-req: 0 [2026-09-22 02:55:31 TP0] Decode batch, #running-req: 1, #full token: 2695, full token usage: 0.03, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.33, #queue-req: 0 [2026-09-22 02:55:32 TP0] Decode batch, #running-req: 1, #full token: 2735, full token usage: 0.03, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.12, #queue-req: 0 [2026-09-22 02:55:32 TP0] Decode batch, #running-req: 1, #full token: 2775, full token usage: 0.03, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.18, #queue-req: 0 [2026-09-22 02:55:32 TP0] Decode batch, #running-req: 1, #full token: 2815, full token usage: 0.03, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.13, #queue-req: 0 [2026-09-22 02:55:33 TP0] Decode batch, #running-req: 1, #full token: 2855, full token usage: 0.03, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.13, #queue-req: 0 [2026-09-22 02:55:33 TP0] Decode batch, #running-req: 1, #full token: 2895, full token usage: 0.03, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.97, #queue-req: 0 [2026-09-22 02:55:33 TP0] Decode batch, #running-req: 1, #full token: 2935, full token usage: 0.03, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.01, #queue-req: 0 [2026-09-22 02:55:34 TP0] Decode batch, #running-req: 1, #full token: 2975, full token usage: 0.03, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.04, #queue-req: 0 [2026-09-22 02:55:34 TP0] Decode batch, #running-req: 1, #full token: 3015, full token usage: 0.03, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.57, #queue-req: 0 [2026-09-22 02:55:34 TP0] Decode batch, #running-req: 1, #full token: 3055, full token usage: 0.03, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.28, #queue-req: 0 [2026-09-22 02:55:35 TP0] Decode batch, #running-req: 1, #full token: 3095, full token usage: 0.03, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.85, #queue-req: 0 [2026-09-22 02:55:35 TP0] Decode batch, #running-req: 1, #full token: 3135, full token usage: 0.03, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.67, #queue-req: 0 [2026-09-22 02:55:35 TP0] Decode batch, #running-req: 1, #full token: 3175, full token usage: 0.03, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.17, #queue-req: 0 [2026-09-22 02:55:35 TP0] Decode batch, #running-req: 1, #full token: 3215, full token usage: 0.04, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.54, #queue-req: 0 [2026-09-22 02:55:36 TP0] Decode batch, #running-req: 1, #full token: 3255, full token usage: 0.04, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.02, #queue-req: 0 [2026-09-22 02:55:36 TP0] Decode batch, #running-req: 1, #full token: 3295, full token usage: 0.04, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.21, #queue-req: 0 [2026-09-22 02:55:36 TP0] Decode batch, #running-req: 1, #full token: 3335, full token usage: 0.04, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.01, #queue-req: 0 [2026-09-22 02:55:37 TP0] Decode batch, #running-req: 1, #full token: 3375, full token usage: 0.04, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.64, #queue-req: 0 [2026-09-22 02:55:37 TP0] Decode batch, #running-req: 1, #full token: 3415, full token usage: 0.04, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.67, #queue-req: 0 [2026-09-22 02:55:37 TP0] Decode batch, #running-req: 1, #full token: 3455, full token usage: 0.04, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.76, #queue-req: 0 [2026-09-22 02:55:38 TP0] Decode batch, #running-req: 1, #full token: 3495, full token usage: 0.04, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.42, #queue-req: 0 [2026-09-22 02:55:38 TP0] Decode batch, #running-req: 1, #full token: 3535, full token usage: 0.04, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.38, #queue-req: 0 [2026-09-22 02:55:38 TP0] Decode batch, #running-req: 1, #full token: 3575, full token usage: 0.04, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.47, #queue-req: 0 [2026-09-22 02:55:39 TP0] Decode batch, #running-req: 1, #full token: 3615, full token usage: 0.04, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.57, #queue-req: 0 [2026-09-22 02:55:39 TP0] Decode batch, #running-req: 1, #full token: 3655, full token usage: 0.04, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.53, #queue-req: 0 [2026-09-22 02:55:39 TP0] Decode batch, #running-req: 1, #full token: 3695, full token usage: 0.04, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.22, #queue-req: 0 [2026-09-22 02:55:40 TP0] Decode batch, #running-req: 1, #full token: 3735, full token usage: 0.04, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.77, #queue-req: 0 [2026-09-22 02:55:40 TP0] Decode batch, #running-req: 1, #full token: 3775, full token usage: 0.04, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.77, #queue-req: 0 [2026-09-22 02:55:40 TP0] Decode batch, #running-req: 1, #full token: 3815, full token usage: 0.04, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.79, #queue-req: 0 [2026-09-22 02:55:41 TP0] Decode batch, #running-req: 1, #full token: 3855, full token usage: 0.04, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.24, #queue-req: 0 [2026-09-22 02:55:41 TP0] Decode batch, #running-req: 1, #full token: 3895, full token usage: 0.04, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.26, #queue-req: 0 [2026-09-22 02:55:41 TP0] Decode batch, #running-req: 1, #full token: 3935, full token usage: 0.04, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.35, #queue-req: 0 [2026-09-22 02:55:42 TP0] Decode batch, #running-req: 1, #full token: 3975, full token usage: 0.04, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.45, #queue-req: 0 [2026-09-22 02:55:42 TP0] Decode batch, #running-req: 1, #full token: 4015, full token usage: 0.04, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.49, #queue-req: 0 [2026-09-22 02:55:42 TP0] Decode batch, #running-req: 1, #full token: 4055, full token usage: 0.04, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.95, #queue-req: 0 [2026-09-22 02:55:42 TP0] Decode batch, #running-req: 1, #full token: 4095, full token usage: 0.04, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.76, #queue-req: 0 [2026-09-22 02:55:43 TP0] Decode batch, #running-req: 1, #full token: 4135, full token usage: 0.05, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.82, #queue-req: 0 [2026-09-22 02:55:43 TP0] Decode batch, #running-req: 1, #full token: 4175, full token usage: 0.05, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.81, #queue-req: 0 [2026-09-22 02:55:43 TP0] Decode batch, #running-req: 1, #full token: 4215, full token usage: 0.05, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.13, #queue-req: 0 [2026-09-22 02:55:44 TP0] Decode batch, #running-req: 1, #full token: 4255, full token usage: 0.05, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.86, #queue-req: 0 [2026-09-22 02:55:44 TP0] Decode batch, #running-req: 1, #full token: 4295, full token usage: 0.05, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.69, #queue-req: 0 [2026-09-22 02:55:44 TP0] Decode batch, #running-req: 1, #full token: 4335, full token usage: 0.05, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.52, #queue-req: 0 [2026-09-22 02:55:45 TP0] Decode batch, #running-req: 1, #full token: 4375, full token usage: 0.05, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.00, #queue-req: 0 [2026-09-22 02:55:45 TP0] Decode batch, #running-req: 1, #full token: 4415, full token usage: 0.05, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.22, #queue-req: 0 [2026-09-22 02:55:45 TP0] Decode batch, #running-req: 1, #full token: 4455, full token usage: 0.05, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.17, #queue-req: 0 [2026-09-22 02:55:46 TP0] Decode batch, #running-req: 1, #full token: 4495, full token usage: 0.05, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.65, #queue-req: 0 [2026-09-22 02:55:46 TP0] Decode batch, #running-req: 1, #full token: 4535, full token usage: 0.05, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.15, #queue-req: 0 [2026-09-22 02:55:46 TP0] Decode batch, #running-req: 1, #full token: 4575, full token usage: 0.05, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.07, #queue-req: 0 [2026-09-22 02:55:47 TP0] Decode batch, #running-req: 1, #full token: 4615, full token usage: 0.05, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.17, #queue-req: 0 [2026-09-22 02:55:47 TP0] Decode batch, #running-req: 1, #full token: 4655, full token usage: 0.05, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.03, #queue-req: 0 [2026-09-22 02:55:47 TP0] Decode batch, #running-req: 1, #full token: 4695, full token usage: 0.05, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.03, #queue-req: 0 [2026-09-22 02:55:48 TP0] Decode batch, #running-req: 1, #full token: 4735, full token usage: 0.05, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.13, #queue-req: 0 [2026-09-22 02:55:48 TP0] Decode batch, #running-req: 1, #full token: 4775, full token usage: 0.05, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.85, #queue-req: 0 [2026-09-22 02:55:48 TP0] Decode batch, #running-req: 1, #full token: 4815, full token usage: 0.05, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.30, #queue-req: 0 [2026-09-22 02:55:49 TP0] Decode batch, #running-req: 1, #full token: 4855, full token usage: 0.05, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.26, #queue-req: 0 [2026-09-22 02:55:49 TP0] Decode batch, #running-req: 1, #full token: 4895, full token usage: 0.05, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.65, #queue-req: 0 [2026-09-22 02:55:49 TP0] Decode batch, #running-req: 1, #full token: 4935, full token usage: 0.05, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.49, #queue-req: 0 [2026-09-22 02:55:50 TP0] Decode batch, #running-req: 1, #full token: 4975, full token usage: 0.05, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.49, #queue-req: 0 [2026-09-22 02:55:50 TP0] Decode batch, #running-req: 1, #full token: 5015, full token usage: 0.05, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.46, #queue-req: 0 [2026-09-22 02:55:50 TP0] Decode batch, #running-req: 1, #full token: 5055, full token usage: 0.06, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.37, #queue-req: 0 [2026-09-22 02:55:51 TP0] Decode batch, #running-req: 1, #full token: 5095, full token usage: 0.06, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.78, #queue-req: 0 [2026-09-22 02:55:51 TP0] Decode batch, #running-req: 1, #full token: 5135, full token usage: 0.06, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.79, #queue-req: 0 [2026-09-22 02:55:51 TP0] Decode batch, #running-req: 1, #full token: 5175, full token usage: 0.06, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.27, #queue-req: 0 [2026-09-22 02:55:51 TP0] Decode batch, #running-req: 1, #full token: 5215, full token usage: 0.06, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.07, #queue-req: 0 [2026-09-22 02:55:52 TP0] Decode batch, #running-req: 1, #full token: 5255, full token usage: 0.06, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.42, #queue-req: 0 [2026-09-22 02:55:52 TP0] Decode batch, #running-req: 1, #full token: 5295, full token usage: 0.06, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.16, #queue-req: 0 [2026-09-22 02:55:52 TP0] Decode batch, #running-req: 1, #full token: 5335, full token usage: 0.06, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.27, #queue-req: 0 [2026-09-22 02:55:53 TP0] Decode batch, #running-req: 1, #full token: 5375, full token usage: 0.06, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.86, #queue-req: 0 [2026-09-22 02:55:53 TP0] Decode batch, #running-req: 1, #full token: 5415, full token usage: 0.06, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.82, #queue-req: 0 [2026-09-22 02:55:53 TP0] Decode batch, #running-req: 1, #full token: 5455, full token usage: 0.06, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.72, #queue-req: 0 [2026-09-22 02:55:54 TP0] Decode batch, #running-req: 1, #full token: 5495, full token usage: 0.06, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.96, #queue-req: 0 [2026-09-22 02:55:54 TP0] Decode batch, #running-req: 1, #full token: 5535, full token usage: 0.06, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.18, #queue-req: 0 [2026-09-22 02:55:54 TP0] Decode batch, #running-req: 1, #full token: 5575, full token usage: 0.06, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.08, #queue-req: 0 [2026-09-22 02:55:55 TP0] Decode batch, #running-req: 1, #full token: 5615, full token usage: 0.06, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.29, #queue-req: 0 [2026-09-22 02:55:55 TP0] Decode batch, #running-req: 1, #full token: 5655, full token usage: 0.06, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.76, #queue-req: 0 [2026-09-22 02:55:55 TP0] Decode batch, #running-req: 1, #full token: 5695, full token usage: 0.06, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.24, #queue-req: 0 [2026-09-22 02:55:56 TP0] Decode batch, #running-req: 1, #full token: 5735, full token usage: 0.06, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.51, #queue-req: 0 [2026-09-22 02:55:56 TP0] Decode batch, #running-req: 1, #full token: 5775, full token usage: 0.06, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.75, #queue-req: 0 [2026-09-22 02:55:56 TP0] Decode batch, #running-req: 1, #full token: 5815, full token usage: 0.06, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.14, #queue-req: 0 [2026-09-22 02:55:57 TP0] Decode batch, #running-req: 1, #full token: 5855, full token usage: 0.06, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.96, #queue-req: 0 [2026-09-22 02:55:57 TP0] Decode batch, #running-req: 1, #full token: 5895, full token usage: 0.06, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.46, #queue-req: 0 [2026-09-22 02:55:57 TP0] Decode batch, #running-req: 1, #full token: 5935, full token usage: 0.06, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.76, #queue-req: 0 [2026-09-22 02:55:58 TP0] Decode batch, #running-req: 1, #full token: 5975, full token usage: 0.07, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.11, #queue-req: 0 [2026-09-22 02:55:58 TP0] Decode batch, #running-req: 1, #full token: 6015, full token usage: 0.07, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.63, #queue-req: 0 [2026-09-22 02:55:58 TP0] Decode batch, #running-req: 1, #full token: 6055, full token usage: 0.07, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.66, #queue-req: 0 [2026-09-22 02:55:59 TP0] Decode batch, #running-req: 1, #full token: 6095, full token usage: 0.07, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.58, #queue-req: 0 [2026-09-22 02:55:59 TP0] Decode batch, #running-req: 1, #full token: 6135, full token usage: 0.07, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.50, #queue-req: 0 [2026-09-22 02:55:59 TP0] Decode batch, #running-req: 1, #full token: 6175, full token usage: 0.07, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.14, #queue-req: 0 [2026-09-22 02:56:00 TP0] Decode batch, #running-req: 1, #full token: 6215, full token usage: 0.07, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.73, #queue-req: 0 [2026-09-22 02:56:00 TP0] Decode batch, #running-req: 1, #full token: 6255, full token usage: 0.07, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.82, #queue-req: 0 [2026-09-22 02:56:00 TP0] Decode batch, #running-req: 1, #full token: 6295, full token usage: 0.07, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.54, #queue-req: 0 [2026-09-22 02:56:01 TP0] Decode batch, #running-req: 1, #full token: 6335, full token usage: 0.07, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.99, #queue-req: 0 [2026-09-22 02:56:01 TP0] Decode batch, #running-req: 1, #full token: 6375, full token usage: 0.07, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.10, #queue-req: 0 [2026-09-22 02:56:01 TP0] Decode batch, #running-req: 1, #full token: 6415, full token usage: 0.07, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.06, #queue-req: 0 [2026-09-22 02:56:02 TP0] Decode batch, #running-req: 1, #full token: 6455, full token usage: 0.07, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.60, #queue-req: 0 [2026-09-22 02:56:02 TP0] Decode batch, #running-req: 1, #full token: 6495, full token usage: 0.07, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.72, #queue-req: 0 [2026-09-22 02:56:02 TP0] Decode batch, #running-req: 1, #full token: 6535, full token usage: 0.07, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.93, #queue-req: 0 [2026-09-22 02:56:03 TP0] Decode batch, #running-req: 1, #full token: 6575, full token usage: 0.07, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.78, #queue-req: 0 [2026-09-22 02:56:03 TP0] Decode batch, #running-req: 1, #full token: 6615, full token usage: 0.07, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.27, #queue-req: 0 [2026-09-22 02:56:03 TP0] Decode batch, #running-req: 1, #full token: 6655, full token usage: 0.07, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.20, #queue-req: 0 [2026-09-22 02:56:04 TP0] Decode batch, #running-req: 1, #full token: 6695, full token usage: 0.07, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.77, #queue-req: 0 [2026-09-22 02:56:04 TP0] Decode batch, #running-req: 1, #full token: 6735, full token usage: 0.07, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.21, #queue-req: 0 [2026-09-22 02:56:04 TP0] Decode batch, #running-req: 1, #full token: 6775, full token usage: 0.07, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.51, #queue-req: 0 [2026-09-22 02:56:04 TP0] Decode batch, #running-req: 1, #full token: 6815, full token usage: 0.07, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.66, #queue-req: 0 [2026-09-22 02:56:05 TP0] Decode batch, #running-req: 1, #full token: 6855, full token usage: 0.08, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.66, #queue-req: 0 [2026-09-22 02:56:05 TP0] Decode batch, #running-req: 1, #full token: 6895, full token usage: 0.08, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.91, #queue-req: 0 [2026-09-22 02:56:05 TP0] Decode batch, #running-req: 1, #full token: 6935, full token usage: 0.08, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.77, #queue-req: 0 [2026-09-22 02:56:06 TP0] Decode batch, #running-req: 1, #full token: 6975, full token usage: 0.08, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.66, #queue-req: 0 [2026-09-22 02:56:06 TP0] Decode batch, #running-req: 1, #full token: 7015, full token usage: 0.08, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.59, #queue-req: 0 [2026-09-22 02:56:06 TP0] Decode batch, #running-req: 1, #full token: 7055, full token usage: 0.08, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.45, #queue-req: 0 [2026-09-22 02:56:07 TP0] Decode batch, #running-req: 1, #full token: 7095, full token usage: 0.08, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.95, #queue-req: 0 [2026-09-22 02:56:07 TP0] Decode batch, #running-req: 1, #full token: 7135, full token usage: 0.08, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.53, #queue-req: 0 [2026-09-22 02:56:07 TP0] Decode batch, #running-req: 1, #full token: 7175, full token usage: 0.08, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.55, #queue-req: 0 [2026-09-22 02:56:08 TP0] Decode batch, #running-req: 1, #full token: 7215, full token usage: 0.08, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.35, #queue-req: 0 [2026-09-22 02:56:08 TP0] Decode batch, #running-req: 1, #full token: 7255, full token usage: 0.08, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.88, #queue-req: 0 [2026-09-22 02:56:08 TP0] Decode batch, #running-req: 1, #full token: 7295, full token usage: 0.08, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.62, #queue-req: 0 [2026-09-22 02:56:09 TP0] Decode batch, #running-req: 1, #full token: 7335, full token usage: 0.08, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.36, #queue-req: 0 [2026-09-22 02:56:09 TP0] Decode batch, #running-req: 1, #full token: 7375, full token usage: 0.08, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.04, #queue-req: 0 [2026-09-22 02:56:09 TP0] Decode batch, #running-req: 1, #full token: 7415, full token usage: 0.08, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.53, #queue-req: 0 [2026-09-22 02:56:10 TP0] Decode batch, #running-req: 1, #full token: 7455, full token usage: 0.08, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.19, #queue-req: 0 [2026-09-22 02:56:10 TP0] Decode batch, #running-req: 1, #full token: 7495, full token usage: 0.08, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.52, #queue-req: 0 [2026-09-22 02:56:10 TP0] Decode batch, #running-req: 1, #full token: 7535, full token usage: 0.08, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.42, #queue-req: 0 [2026-09-22 02:56:11 TP0] Decode batch, #running-req: 1, #full token: 7575, full token usage: 0.08, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.08, #queue-req: 0 [2026-09-22 02:56:11 TP0] Decode batch, #running-req: 1, #full token: 7615, full token usage: 0.08, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.04, #queue-req: 0 [2026-09-22 02:56:11 TP0] Decode batch, #running-req: 1, #full token: 7655, full token usage: 0.08, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.78, #queue-req: 0 [2026-09-22 02:56:12 TP0] Decode batch, #running-req: 1, #full token: 7695, full token usage: 0.08, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.87, #queue-req: 0 [2026-09-22 02:56:12 TP0] Decode batch, #running-req: 1, #full token: 7735, full token usage: 0.08, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.81, #queue-req: 0 [2026-09-22 02:56:12 TP0] Decode batch, #running-req: 1, #full token: 7775, full token usage: 0.09, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.12, #queue-req: 0 [2026-09-22 02:56:13 TP0] Decode batch, #running-req: 1, #full token: 7815, full token usage: 0.09, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.15, #queue-req: 0 [2026-09-22 02:56:13 TP0] Decode batch, #running-req: 1, #full token: 7855, full token usage: 0.09, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.10, #queue-req: 0 [2026-09-22 02:56:13 TP0] Decode batch, #running-req: 1, #full token: 7895, full token usage: 0.09, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.25, #queue-req: 0 [2026-09-22 02:56:14 TP0] Decode batch, #running-req: 1, #full token: 7935, full token usage: 0.09, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.20, #queue-req: 0 [2026-09-22 02:56:14 TP0] Decode batch, #running-req: 1, #full token: 7975, full token usage: 0.09, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.73, #queue-req: 0 [2026-09-22 02:56:14 TP0] Decode batch, #running-req: 1, #full token: 8015, full token usage: 0.09, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.99, #queue-req: 0 [2026-09-22 02:56:15 TP0] Decode batch, #running-req: 1, #full token: 8055, full token usage: 0.09, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.04, #queue-req: 0 [2026-09-22 02:56:15 TP0] Decode batch, #running-req: 1, #full token: 8095, full token usage: 0.09, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.73, #queue-req: 0 [2026-09-22 02:56:15 TP0] Decode batch, #running-req: 1, #full token: 8135, full token usage: 0.09, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.16, #queue-req: 0 [2026-09-22 02:56:16 TP0] Decode batch, #running-req: 1, #full token: 8175, full token usage: 0.09, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.27, #queue-req: 0 [2026-09-22 02:56:16 TP0] Decode batch, #running-req: 1, #full token: 8215, full token usage: 0.09, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.27, #queue-req: 0 [2026-09-22 02:56:16 TP0] Decode batch, #running-req: 1, #full token: 8255, full token usage: 0.09, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.05, #queue-req: 0 [2026-09-22 02:56:17 TP0] Decode batch, #running-req: 1, #full token: 8295, full token usage: 0.09, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.10, #queue-req: 0 [2026-09-22 02:56:17 TP0] Decode batch, #running-req: 1, #full token: 8335, full token usage: 0.09, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.06, #queue-req: 0 [2026-09-22 02:56:17 TP0] Decode batch, #running-req: 1, #full token: 8375, full token usage: 0.09, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.98, #queue-req: 0 [2026-09-22 02:56:18 TP0] Decode batch, #running-req: 1, #full token: 8415, full token usage: 0.09, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.05, #queue-req: 0 [2026-09-22 02:56:18 TP0] Decode batch, #running-req: 1, #full token: 8455, full token usage: 0.09, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.05, #queue-req: 0 [2026-09-22 02:56:18 TP0] Decode batch, #running-req: 1, #full token: 8495, full token usage: 0.09, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.07, #queue-req: 0 [2026-09-22 02:56:19 TP0] Decode batch, #running-req: 1, #full token: 8535, full token usage: 0.09, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.30, #queue-req: 0 [2026-09-22 02:56:19 TP0] Decode batch, #running-req: 1, #full token: 8575, full token usage: 0.09, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.38, #queue-req: 0 [2026-09-22 02:56:19 TP0] Decode batch, #running-req: 1, #full token: 8615, full token usage: 0.09, #swa token: 187, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.25, #queue-req: 0 [2026-09-22 02:56:20 TP0] Decode batch, #running-req: 1, #full token: 8655, full token usage: 0.09, #swa token: 227, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.16, #queue-req: 0 [2026-09-22 02:56:20 TP0] Decode batch, #running-req: 1, #full token: 8695, full token usage: 0.10, #swa token: 267, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.98, #queue-req: 0 [2026-09-22 02:56:20 TP0] Decode batch, #running-req: 1, #full token: 8735, full token usage: 0.10, #swa token: 179, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.56, #queue-req: 0 [2026-09-22 02:56:21 TP0] Decode batch, #running-req: 1, #full token: 8775, full token usage: 0.10, #swa token: 219, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.85, #queue-req: 0 [2026-09-22 02:56:21 TP0] Decode batch, #running-req: 1, #full token: 8815, full token usage: 0.10, #swa token: 259, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.13, #queue-req: 0 [2026-09-22 02:56:21 TP0] Decode batch, #running-req: 1, #full token: 8855, full token usage: 0.10, #swa token: 171, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.57, #queue-req: 0 [2026-09-22 02:56:22 TP0] Decode batch, #running-req: 1, #full token: 8895, full token usage: 0.10, #swa token: 211, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.03, #queue-req: 0 [2026-09-22 02:56:22 TP0] Decode batch, #running-req: 1, #full token: 8935, full token usage: 0.10, #swa token: 251, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.99, #queue-req: 0 [2026-09-22 02:56:22 TP0] Decode batch, #running-req: 1, #full token: 8975, full token usage: 0.10, #swa token: 163, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.68, #queue-req: 0 [2026-09-22 02:56:23 TP0] Decode batch, #running-req: 1, #full token: 9015, full token usage: 0.10, #swa token: 203, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.69, #queue-req: 0 [2026-09-22 02:56:23 TP0] Decode batch, #running-req: 1, #full token: 9055, full token usage: 0.10, #swa token: 243, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.80, #queue-req: 0 [2026-09-22 02:56:23 TP0] Decode batch, #running-req: 1, #full token: 9095, full token usage: 0.10, #swa token: 155, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.71, #queue-req: 0 [2026-09-22 02:56:24 TP0] Decode batch, #running-req: 1, #full token: 9135, full token usage: 0.10, #swa token: 195, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.39, #queue-req: 0 [2026-09-22 02:56:24 TP0] Decode batch, #running-req: 1, #full token: 9175, full token usage: 0.10, #swa token: 235, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.40, #queue-req: 0 [2026-09-22 02:56:24 TP0] Decode batch, #running-req: 1, #full token: 9215, full token usage: 0.10, #swa token: 275, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.29, #queue-req: 0 [2026-09-22 02:56:25 TP0] Prefill batch, #new-seq: 1, #new-token: 22, #cached-token: 3, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.30 [2026-09-22 02:56:25 TP0] Decode batch, #running-req: 1, #full token: 41, full token usage: 0.00, #swa token: 41, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 111.41, #queue-req: 0 [2026-09-22 02:56:25 TP0] Decode batch, #running-req: 1, #full token: 81, full token usage: 0.00, #swa token: 81, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.53, #queue-req: 0 [2026-09-22 02:56:25 TP0] Decode batch, #running-req: 1, #full token: 121, full token usage: 0.00, #swa token: 121, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.18, #queue-req: 0 [2026-09-22 02:56:26 TP0] Decode batch, #running-req: 1, #full token: 161, full token usage: 0.00, #swa token: 161, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.85, #queue-req: 0 [2026-09-22 02:56:26 TP0] Decode batch, #running-req: 1, #full token: 201, full token usage: 0.00, #swa token: 201, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.48, #queue-req: 0 [2026-09-22 02:56:26 TP0] Decode batch, #running-req: 1, #full token: 241, full token usage: 0.00, #swa token: 241, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.36, #queue-req: 0 [2026-09-22 02:56:27 TP0] Decode batch, #running-req: 1, #full token: 281, full token usage: 0.00, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 130.03, #queue-req: 0 [2026-09-22 02:56:27 TP0] Decode batch, #running-req: 1, #full token: 321, full token usage: 0.00, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.78, #queue-req: 0 [2026-09-22 02:56:27 TP0] Decode batch, #running-req: 1, #full token: 361, full token usage: 0.00, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.27, #queue-req: 0 [2026-09-22 02:56:28 TP0] Decode batch, #running-req: 1, #full token: 401, full token usage: 0.00, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.37, #queue-req: 0 [2026-09-22 02:56:28 TP0] Decode batch, #running-req: 1, #full token: 441, full token usage: 0.00, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.26, #queue-req: 0 [2026-09-22 02:56:28 TP0] Decode batch, #running-req: 1, #full token: 481, full token usage: 0.01, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.35, #queue-req: 0 [2026-09-22 02:56:28 TP0] Decode batch, #running-req: 1, #full token: 521, full token usage: 0.01, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.32, #queue-req: 0 [2026-09-22 02:56:29 TP0] Decode batch, #running-req: 1, #full token: 561, full token usage: 0.01, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.47, #queue-req: 0 [2026-09-22 02:56:29 TP0] Decode batch, #running-req: 1, #full token: 601, full token usage: 0.01, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.85, #queue-req: 0 [2026-09-22 02:56:29 TP0] Decode batch, #running-req: 1, #full token: 641, full token usage: 0.01, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.77, #queue-req: 0 [2026-09-22 02:56:30 TP0] Decode batch, #running-req: 1, #full token: 681, full token usage: 0.01, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.75, #queue-req: 0 [2026-09-22 02:56:30 TP0] Decode batch, #running-req: 1, #full token: 721, full token usage: 0.01, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.79, #queue-req: 0 [2026-09-22 02:56:30 TP0] Decode batch, #running-req: 1, #full token: 761, full token usage: 0.01, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.60, #queue-req: 0 [2026-09-22 02:56:31 TP0] Decode batch, #running-req: 1, #full token: 801, full token usage: 0.01, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.89, #queue-req: 0 [2026-09-22 02:56:31 TP0] Decode batch, #running-req: 1, #full token: 841, full token usage: 0.01, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.37, #queue-req: 0 [2026-09-22 02:56:31 TP0] Decode batch, #running-req: 1, #full token: 881, full token usage: 0.01, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.51, #queue-req: 0 [2026-09-22 02:56:32 TP0] Decode batch, #running-req: 1, #full token: 921, full token usage: 0.01, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.61, #queue-req: 0 [2026-09-22 02:56:32 TP0] Decode batch, #running-req: 1, #full token: 961, full token usage: 0.01, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.65, #queue-req: 0 [2026-09-22 02:56:32 TP0] Decode batch, #running-req: 1, #full token: 1001, full token usage: 0.01, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.51, #queue-req: 0 [2026-09-22 02:56:32 TP0] Decode batch, #running-req: 1, #full token: 1041, full token usage: 0.01, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.49, #queue-req: 0 [2026-09-22 02:56:33 TP0] Decode batch, #running-req: 1, #full token: 1081, full token usage: 0.01, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.20, #queue-req: 0 [2026-09-22 02:56:33 TP0] Decode batch, #running-req: 1, #full token: 1121, full token usage: 0.01, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.15, #queue-req: 0 [2026-09-22 02:56:33 TP0] Decode batch, #running-req: 1, #full token: 1161, full token usage: 0.01, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.24, #queue-req: 0 [2026-09-22 02:56:34 TP0] Decode batch, #running-req: 1, #full token: 1201, full token usage: 0.01, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 129.19, #queue-req: 0 [2026-09-22 02:56:34 TP0] Decode batch, #running-req: 1, #full token: 1241, full token usage: 0.01, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.90, #queue-req: 0 [2026-09-22 02:56:34 TP0] Decode batch, #running-req: 1, #full token: 1281, full token usage: 0.01, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.13, #queue-req: 0 [2026-09-22 02:56:35 TP0] Decode batch, #running-req: 1, #full token: 1321, full token usage: 0.01, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.69, #queue-req: 0 [2026-09-22 02:56:35 TP0] Decode batch, #running-req: 1, #full token: 1361, full token usage: 0.01, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.09, #queue-req: 0 [2026-09-22 02:56:35 TP0] Decode batch, #running-req: 1, #full token: 1401, full token usage: 0.02, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.10, #queue-req: 0 [2026-09-22 02:56:36 TP0] Decode batch, #running-req: 1, #full token: 1441, full token usage: 0.02, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.21, #queue-req: 0 [2026-09-22 02:56:36 TP0] Decode batch, #running-req: 1, #full token: 1481, full token usage: 0.02, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.07, #queue-req: 0 [2026-09-22 02:56:36 TP0] Decode batch, #running-req: 1, #full token: 1521, full token usage: 0.02, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.77, #queue-req: 0 [2026-09-22 02:56:37 TP0] Decode batch, #running-req: 1, #full token: 1561, full token usage: 0.02, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.49, #queue-req: 0 [2026-09-22 02:56:37 TP0] Decode batch, #running-req: 1, #full token: 1601, full token usage: 0.02, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.62, #queue-req: 0 [2026-09-22 02:56:37 TP0] Decode batch, #running-req: 1, #full token: 1641, full token usage: 0.02, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.95, #queue-req: 0 [2026-09-22 02:56:37 TP0] Decode batch, #running-req: 1, #full token: 1681, full token usage: 0.02, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.51, #queue-req: 0 [2026-09-22 02:56:38 TP0] Decode batch, #running-req: 1, #full token: 1721, full token usage: 0.02, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.30, #queue-req: 0 [2026-09-22 02:56:38 TP0] Decode batch, #running-req: 1, #full token: 1761, full token usage: 0.02, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.59, #queue-req: 0 [2026-09-22 02:56:38 TP0] Decode batch, #running-req: 1, #full token: 1801, full token usage: 0.02, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.34, #queue-req: 0 [2026-09-22 02:56:39 TP0] Decode batch, #running-req: 1, #full token: 1841, full token usage: 0.02, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.03, #queue-req: 0 [2026-09-22 02:56:39 TP0] Decode batch, #running-req: 1, #full token: 1881, full token usage: 0.02, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.78, #queue-req: 0 [2026-09-22 02:56:39 TP0] Decode batch, #running-req: 1, #full token: 1921, full token usage: 0.02, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.78, #queue-req: 0 [2026-09-22 02:56:40 TP0] Decode batch, #running-req: 1, #full token: 1961, full token usage: 0.02, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.45, #queue-req: 0 [2026-09-22 02:56:40 TP0] Decode batch, #running-req: 1, #full token: 2001, full token usage: 0.02, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.86, #queue-req: 0 [2026-09-22 02:56:40 TP0] Decode batch, #running-req: 1, #full token: 2041, full token usage: 0.02, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.94, #queue-req: 0 [2026-09-22 02:56:41 TP0] Decode batch, #running-req: 1, #full token: 2081, full token usage: 0.02, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.74, #queue-req: 0 [2026-09-22 02:56:41 TP0] Decode batch, #running-req: 1, #full token: 2121, full token usage: 0.02, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.23, #queue-req: 0 [2026-09-22 02:56:41 TP0] Decode batch, #running-req: 1, #full token: 2161, full token usage: 0.02, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.23, #queue-req: 0 [2026-09-22 02:56:42 TP0] Decode batch, #running-req: 1, #full token: 2201, full token usage: 0.02, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.03, #queue-req: 0 [2026-09-22 02:56:42 TP0] Decode batch, #running-req: 1, #full token: 2241, full token usage: 0.02, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.99, #queue-req: 0 [2026-09-22 02:56:42 TP0] Decode batch, #running-req: 1, #full token: 2281, full token usage: 0.02, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.85, #queue-req: 0 [2026-09-22 02:56:42 TP0] Decode batch, #running-req: 1, #full token: 2321, full token usage: 0.03, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.94, #queue-req: 0 [2026-09-22 02:56:43 TP0] Decode batch, #running-req: 1, #full token: 2361, full token usage: 0.03, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.25, #queue-req: 0 [2026-09-22 02:56:43 TP0] Decode batch, #running-req: 1, #full token: 2401, full token usage: 0.03, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.52, #queue-req: 0 [2026-09-22 02:56:43 TP0] Decode batch, #running-req: 1, #full token: 2441, full token usage: 0.03, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.48, #queue-req: 0 [2026-09-22 02:56:44 TP0] Decode batch, #running-req: 1, #full token: 2481, full token usage: 0.03, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.30, #queue-req: 0 [2026-09-22 02:56:44 TP0] Decode batch, #running-req: 1, #full token: 2521, full token usage: 0.03, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.23, #queue-req: 0 [2026-09-22 02:56:44 TP0] Decode batch, #running-req: 1, #full token: 2561, full token usage: 0.03, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.39, #queue-req: 0 [2026-09-22 02:56:45 TP0] Decode batch, #running-req: 1, #full token: 2601, full token usage: 0.03, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.09, #queue-req: 0 [2026-09-22 02:56:45 TP0] Decode batch, #running-req: 1, #full token: 2641, full token usage: 0.03, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.18, #queue-req: 0 [2026-09-22 02:56:45 TP0] Decode batch, #running-req: 1, #full token: 2681, full token usage: 0.03, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.14, #queue-req: 0 [2026-09-22 02:56:46 TP0] Decode batch, #running-req: 1, #full token: 2721, full token usage: 0.03, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.23, #queue-req: 0 [2026-09-22 02:56:46 TP0] Decode batch, #running-req: 1, #full token: 2761, full token usage: 0.03, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.78, #queue-req: 0 [2026-09-22 02:56:46 TP0] Decode batch, #running-req: 1, #full token: 2801, full token usage: 0.03, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.23, #queue-req: 0 [2026-09-22 02:56:47 TP0] Decode batch, #running-req: 1, #full token: 2841, full token usage: 0.03, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.09, #queue-req: 0 [2026-09-22 02:56:47 TP0] Decode batch, #running-req: 1, #full token: 2881, full token usage: 0.03, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.14, #queue-req: 0 [2026-09-22 02:56:47 TP0] Decode batch, #running-req: 1, #full token: 2921, full token usage: 0.03, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.17, #queue-req: 0 [2026-09-22 02:56:47 TP0] Decode batch, #running-req: 1, #full token: 2961, full token usage: 0.03, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.04, #queue-req: 0 [2026-09-22 02:56:48 TP0] Decode batch, #running-req: 1, #full token: 3001, full token usage: 0.03, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.15, #queue-req: 0 [2026-09-22 02:56:48 TP0] Decode batch, #running-req: 1, #full token: 3041, full token usage: 0.03, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.70, #queue-req: 0 [2026-09-22 02:56:48 TP0] Decode batch, #running-req: 1, #full token: 3081, full token usage: 0.03, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.75, #queue-req: 0 [2026-09-22 02:56:49 TP0] Decode batch, #running-req: 1, #full token: 3121, full token usage: 0.03, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.16, #queue-req: 0 [2026-09-22 02:56:49 TP0] Decode batch, #running-req: 1, #full token: 3161, full token usage: 0.03, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.20, #queue-req: 0 [2026-09-22 02:56:49 TP0] Decode batch, #running-req: 1, #full token: 3201, full token usage: 0.04, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.91, #queue-req: 0 [2026-09-22 02:56:50 TP0] Decode batch, #running-req: 1, #full token: 3241, full token usage: 0.04, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.85, #queue-req: 0 [2026-09-22 02:56:50 TP0] Decode batch, #running-req: 1, #full token: 3281, full token usage: 0.04, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.48, #queue-req: 0 [2026-09-22 02:56:50 TP0] Decode batch, #running-req: 1, #full token: 3321, full token usage: 0.04, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.19, #queue-req: 0 [2026-09-22 02:56:51 TP0] Decode batch, #running-req: 1, #full token: 3361, full token usage: 0.04, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.61, #queue-req: 0 [2026-09-22 02:56:51 TP0] Decode batch, #running-req: 1, #full token: 3401, full token usage: 0.04, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.35, #queue-req: 0 [2026-09-22 02:56:51 TP0] Decode batch, #running-req: 1, #full token: 3441, full token usage: 0.04, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.49, #queue-req: 0 [2026-09-22 02:56:52 TP0] Decode batch, #running-req: 1, #full token: 3481, full token usage: 0.04, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.51, #queue-req: 0 [2026-09-22 02:56:52 TP0] Decode batch, #running-req: 1, #full token: 3521, full token usage: 0.04, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.47, #queue-req: 0 [2026-09-22 02:56:52 TP0] Decode batch, #running-req: 1, #full token: 3561, full token usage: 0.04, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.40, #queue-req: 0 [2026-09-22 02:56:53 TP0] Decode batch, #running-req: 1, #full token: 3601, full token usage: 0.04, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.12, #queue-req: 0 [2026-09-22 02:56:53 TP0] Decode batch, #running-req: 1, #full token: 3641, full token usage: 0.04, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.12, #queue-req: 0 [2026-09-22 02:56:53 TP0] Decode batch, #running-req: 1, #full token: 3681, full token usage: 0.04, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.19, #queue-req: 0 [2026-09-22 02:56:54 TP0] Decode batch, #running-req: 1, #full token: 3721, full token usage: 0.04, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.16, #queue-req: 0 [2026-09-22 02:56:54 TP0] Decode batch, #running-req: 1, #full token: 3761, full token usage: 0.04, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.06, #queue-req: 0 [2026-09-22 02:56:54 TP0] Decode batch, #running-req: 1, #full token: 3801, full token usage: 0.04, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.86, #queue-req: 0 [2026-09-22 02:56:54 TP0] Decode batch, #running-req: 1, #full token: 3841, full token usage: 0.04, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.53, #queue-req: 0 [2026-09-22 02:56:55 TP0] Decode batch, #running-req: 1, #full token: 3881, full token usage: 0.04, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.67, #queue-req: 0 [2026-09-22 02:56:55 TP0] Decode batch, #running-req: 1, #full token: 3921, full token usage: 0.04, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.87, #queue-req: 0 [2026-09-22 02:56:55 TP0] Decode batch, #running-req: 1, #full token: 3961, full token usage: 0.04, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.94, #queue-req: 0 [2026-09-22 02:56:56 TP0] Decode batch, #running-req: 1, #full token: 4001, full token usage: 0.04, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.02, #queue-req: 0 [2026-09-22 02:56:56 TP0] Decode batch, #running-req: 1, #full token: 4041, full token usage: 0.04, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.89, #queue-req: 0 [2026-09-22 02:56:56 TP0] Decode batch, #running-req: 1, #full token: 4081, full token usage: 0.04, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.89, #queue-req: 0 [2026-09-22 02:56:57 TP0] Decode batch, #running-req: 1, #full token: 4121, full token usage: 0.05, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.73, #queue-req: 0 [2026-09-22 02:56:57 TP0] Decode batch, #running-req: 1, #full token: 4161, full token usage: 0.05, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.66, #queue-req: 0 [2026-09-22 02:56:57 TP0] Decode batch, #running-req: 1, #full token: 4201, full token usage: 0.05, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.62, #queue-req: 0 [2026-09-22 02:56:58 TP0] Decode batch, #running-req: 1, #full token: 4241, full token usage: 0.05, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.54, #queue-req: 0 [2026-09-22 02:56:58 TP0] Decode batch, #running-req: 1, #full token: 4281, full token usage: 0.05, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.67, #queue-req: 0 [2026-09-22 02:56:58 TP0] Decode batch, #running-req: 1, #full token: 4321, full token usage: 0.05, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.43, #queue-req: 0 [2026-09-22 02:56:59 TP0] Decode batch, #running-req: 1, #full token: 4361, full token usage: 0.05, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.29, #queue-req: 0 [2026-09-22 02:56:59 TP0] Decode batch, #running-req: 1, #full token: 4401, full token usage: 0.05, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.81, #queue-req: 0 [2026-09-22 02:56:59 TP0] Decode batch, #running-req: 1, #full token: 4441, full token usage: 0.05, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.38, #queue-req: 0 [2026-09-22 02:57:00 TP0] Decode batch, #running-req: 1, #full token: 4481, full token usage: 0.05, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.19, #queue-req: 0 [2026-09-22 02:57:00 TP0] Decode batch, #running-req: 1, #full token: 4521, full token usage: 0.05, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.65, #queue-req: 0 [2026-09-22 02:57:00 TP0] Decode batch, #running-req: 1, #full token: 4561, full token usage: 0.05, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.52, #queue-req: 0 [2026-09-22 02:57:01 TP0] Decode batch, #running-req: 1, #full token: 4601, full token usage: 0.05, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.40, #queue-req: 0 [2026-09-22 02:57:01 TP0] Decode batch, #running-req: 1, #full token: 4641, full token usage: 0.05, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.56, #queue-req: 0 [2026-09-22 02:57:01 TP0] Decode batch, #running-req: 1, #full token: 4681, full token usage: 0.05, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.23, #queue-req: 0 [2026-09-22 02:57:02 TP0] Decode batch, #running-req: 1, #full token: 4721, full token usage: 0.05, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.90, #queue-req: 0 [2026-09-22 02:57:02 TP0] Decode batch, #running-req: 1, #full token: 4761, full token usage: 0.05, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.12, #queue-req: 0 [2026-09-22 02:57:02 TP0] Decode batch, #running-req: 1, #full token: 4801, full token usage: 0.05, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.02, #queue-req: 0 [2026-09-22 02:57:02 TP0] Decode batch, #running-req: 1, #full token: 4841, full token usage: 0.05, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.45, #queue-req: 0 [2026-09-22 02:57:03 TP0] Decode batch, #running-req: 1, #full token: 4881, full token usage: 0.05, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.17, #queue-req: 0 [2026-09-22 02:57:03 TP0] Decode batch, #running-req: 1, #full token: 4921, full token usage: 0.05, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.99, #queue-req: 0 [2026-09-22 02:57:03 TP0] Decode batch, #running-req: 1, #full token: 4961, full token usage: 0.05, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.63, #queue-req: 0 [2026-09-22 02:57:04 TP0] Decode batch, #running-req: 1, #full token: 5001, full token usage: 0.05, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.60, #queue-req: 0 [2026-09-22 02:57:04 TP0] Decode batch, #running-req: 1, #full token: 5041, full token usage: 0.06, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.25, #queue-req: 0 [2026-09-22 02:57:04 TP0] Decode batch, #running-req: 1, #full token: 5081, full token usage: 0.06, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.91, #queue-req: 0 [2026-09-22 02:57:05 TP0] Decode batch, #running-req: 1, #full token: 5121, full token usage: 0.06, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.40, #queue-req: 0 [2026-09-22 02:57:05 TP0] Decode batch, #running-req: 1, #full token: 5161, full token usage: 0.06, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.32, #queue-req: 0 [2026-09-22 02:57:05 TP0] Decode batch, #running-req: 1, #full token: 5201, full token usage: 0.06, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.15, #queue-req: 0 [2026-09-22 02:57:06 TP0] Decode batch, #running-req: 1, #full token: 5241, full token usage: 0.06, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.32, #queue-req: 0 [2026-09-22 02:57:06 TP0] Decode batch, #running-req: 1, #full token: 5281, full token usage: 0.06, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.26, #queue-req: 0 [2026-09-22 02:57:06 TP0] Decode batch, #running-req: 1, #full token: 5321, full token usage: 0.06, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.13, #queue-req: 0 [2026-09-22 02:57:07 TP0] Decode batch, #running-req: 1, #full token: 5361, full token usage: 0.06, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.64, #queue-req: 0 [2026-09-22 02:57:07 TP0] Decode batch, #running-req: 1, #full token: 5401, full token usage: 0.06, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.97, #queue-req: 0 [2026-09-22 02:57:07 TP0] Decode batch, #running-req: 1, #full token: 5441, full token usage: 0.06, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.71, #queue-req: 0 [2026-09-22 02:57:08 TP0] Decode batch, #running-req: 1, #full token: 5481, full token usage: 0.06, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.12, #queue-req: 0 [2026-09-22 02:57:08 TP0] Decode batch, #running-req: 1, #full token: 5521, full token usage: 0.06, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.95, #queue-req: 0 [2026-09-22 02:57:08 TP0] Decode batch, #running-req: 1, #full token: 5561, full token usage: 0.06, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.88, #queue-req: 0 [2026-09-22 02:57:09 TP0] Decode batch, #running-req: 1, #full token: 5601, full token usage: 0.06, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.94, #queue-req: 0 [2026-09-22 02:57:09 TP0] Decode batch, #running-req: 1, #full token: 5641, full token usage: 0.06, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.66, #queue-req: 0 [2026-09-22 02:57:09 TP0] Decode batch, #running-req: 1, #full token: 5681, full token usage: 0.06, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.49, #queue-req: 0 [2026-09-22 02:57:10 TP0] Decode batch, #running-req: 1, #full token: 5721, full token usage: 0.06, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.90, #queue-req: 0 [2026-09-22 02:57:10 TP0] Decode batch, #running-req: 1, #full token: 5761, full token usage: 0.06, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.85, #queue-req: 0 [2026-09-22 02:57:10 TP0] Decode batch, #running-req: 1, #full token: 5801, full token usage: 0.06, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.87, #queue-req: 0 [2026-09-22 02:57:11 TP0] Decode batch, #running-req: 1, #full token: 5841, full token usage: 0.06, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.84, #queue-req: 0 [2026-09-22 02:57:11 TP0] Decode batch, #running-req: 1, #full token: 5881, full token usage: 0.06, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.29, #queue-req: 0 [2026-09-22 02:57:11 TP0] Decode batch, #running-req: 1, #full token: 5921, full token usage: 0.06, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.45, #queue-req: 0 [2026-09-22 02:57:12 TP0] Decode batch, #running-req: 1, #full token: 5961, full token usage: 0.07, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.90, #queue-req: 0 [2026-09-22 02:57:12 TP0] Decode batch, #running-req: 1, #full token: 6001, full token usage: 0.07, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.57, #queue-req: 0 [2026-09-22 02:57:12 TP0] Decode batch, #running-req: 1, #full token: 6041, full token usage: 0.07, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.44, #queue-req: 0 [2026-09-22 02:57:13 TP0] Decode batch, #running-req: 1, #full token: 6081, full token usage: 0.07, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.70, #queue-req: 0 /sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/serving_chat.py:2104: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/ return self.create_error_response(str(e)) [2026-09-22 02:58:16 TP0] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 22, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.01 [2026-09-22 02:58:16 TP0] Decode batch, #running-req: 1, #full token: 54, full token usage: 0.00, #swa token: 54, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 0.63, #queue-req: 0 [2026-09-22 02:58:16 TP0] Decode batch, #running-req: 1, #full token: 94, full token usage: 0.00, #swa token: 94, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.36, #queue-req: 0 [2026-09-22 02:58:17 TP0] Decode batch, #running-req: 1, #full token: 134, full token usage: 0.00, #swa token: 134, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.60, #queue-req: 0 [2026-09-22 02:58:17 TP0] Decode batch, #running-req: 1, #full token: 174, full token usage: 0.00, #swa token: 174, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.32, #queue-req: 0 [2026-09-22 02:58:17 TP0] Decode batch, #running-req: 1, #full token: 214, full token usage: 0.00, #swa token: 214, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.41, #queue-req: 0 [2026-09-22 02:58:17 TP0] Decode batch, #running-req: 1, #full token: 254, full token usage: 0.00, #swa token: 254, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.03, #queue-req: 0 [2026-09-22 02:58:18 TP0] Decode batch, #running-req: 1, #full token: 294, full token usage: 0.00, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.92, #queue-req: 0 [2026-09-22 02:58:18 TP0] Decode batch, #running-req: 1, #full token: 334, full token usage: 0.00, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.15, #queue-req: 0 [2026-09-22 02:58:18 TP0] Decode batch, #running-req: 1, #full token: 374, full token usage: 0.00, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.03, #queue-req: 0 [2026-09-22 02:58:19 TP0] Decode batch, #running-req: 1, #full token: 414, full token usage: 0.00, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.24, #queue-req: 0 [2026-09-22 02:58:19 TP0] Decode batch, #running-req: 1, #full token: 454, full token usage: 0.00, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.44, #queue-req: 0 [2026-09-22 02:58:19 TP0] Decode batch, #running-req: 1, #full token: 494, full token usage: 0.01, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.54, #queue-req: 0 [2026-09-22 02:58:20 TP0] Decode batch, #running-req: 1, #full token: 534, full token usage: 0.01, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.26, #queue-req: 0 [2026-09-22 02:58:20 TP0] Decode batch, #running-req: 1, #full token: 574, full token usage: 0.01, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 128.20, #queue-req: 0 [2026-09-22 02:58:20 TP0] Decode batch, #running-req: 1, #full token: 614, full token usage: 0.01, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.61, #queue-req: 0 [2026-09-22 02:58:21 TP0] Decode batch, #running-req: 1, #full token: 654, full token usage: 0.01, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.81, #queue-req: 0 [2026-09-22 02:58:21 TP0] Decode batch, #running-req: 1, #full token: 694, full token usage: 0.01, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.81, #queue-req: 0 [2026-09-22 02:58:21 TP0] Decode batch, #running-req: 1, #full token: 734, full token usage: 0.01, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.78, #queue-req: 0 [2026-09-22 02:58:22 TP0] Decode batch, #running-req: 1, #full token: 774, full token usage: 0.01, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.46, #queue-req: 0 [2026-09-22 02:58:22 TP0] Decode batch, #running-req: 1, #full token: 814, full token usage: 0.01, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.31, #queue-req: 0 [2026-09-22 02:58:22 TP0] Decode batch, #running-req: 1, #full token: 854, full token usage: 0.01, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.23, #queue-req: 0 [2026-09-22 02:58:22 TP0] Decode batch, #running-req: 1, #full token: 894, full token usage: 0.01, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.22, #queue-req: 0 [2026-09-22 02:58:23 TP0] Decode batch, #running-req: 1, #full token: 934, full token usage: 0.01, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.08, #queue-req: 0 [2026-09-22 02:58:23 TP0] Decode batch, #running-req: 1, #full token: 974, full token usage: 0.01, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.20, #queue-req: 0 [2026-09-22 02:58:23 TP0] Decode batch, #running-req: 1, #full token: 1014, full token usage: 0.01, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.97, #queue-req: 0 [2026-09-22 02:58:24 TP0] Decode batch, #running-req: 1, #full token: 1054, full token usage: 0.01, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.86, #queue-req: 0 [2026-09-22 02:58:24 TP0] Decode batch, #running-req: 1, #full token: 1094, full token usage: 0.01, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.54, #queue-req: 0 [2026-09-22 02:58:24 TP0] Decode batch, #running-req: 1, #full token: 1134, full token usage: 0.01, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.05, #queue-req: 0 [2026-09-22 02:58:25 TP0] Decode batch, #running-req: 1, #full token: 1174, full token usage: 0.01, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.03, #queue-req: 0 [2026-09-22 02:58:25 TP0] Decode batch, #running-req: 1, #full token: 1214, full token usage: 0.01, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.95, #queue-req: 0 [2026-09-22 02:58:25 TP0] Decode batch, #running-req: 1, #full token: 1254, full token usage: 0.01, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 127.17, #queue-req: 0 [2026-09-22 02:58:26 TP0] Decode batch, #running-req: 1, #full token: 1294, full token usage: 0.01, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.89, #queue-req: 0 [2026-09-22 02:58:26 TP0] Decode batch, #running-req: 1, #full token: 1334, full token usage: 0.01, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.65, #queue-req: 0 [2026-09-22 02:58:26 TP0] Decode batch, #running-req: 1, #full token: 1374, full token usage: 0.02, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.76, #queue-req: 0 [2026-09-22 02:58:27 TP0] Decode batch, #running-req: 1, #full token: 1414, full token usage: 0.02, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.62, #queue-req: 0 [2026-09-22 02:58:27 TP0] Decode batch, #running-req: 1, #full token: 1454, full token usage: 0.02, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.97, #queue-req: 0 [2026-09-22 02:58:27 TP0] Decode batch, #running-req: 1, #full token: 1494, full token usage: 0.02, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.87, #queue-req: 0 [2026-09-22 02:58:28 TP0] Decode batch, #running-req: 1, #full token: 1534, full token usage: 0.02, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.62, #queue-req: 0 [2026-09-22 02:58:28 TP0] Decode batch, #running-req: 1, #full token: 1574, full token usage: 0.02, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.64, #queue-req: 0 [2026-09-22 02:58:28 TP0] Decode batch, #running-req: 1, #full token: 1614, full token usage: 0.02, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.70, #queue-req: 0 [2026-09-22 02:58:28 TP0] Decode batch, #running-req: 1, #full token: 1654, full token usage: 0.02, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.83, #queue-req: 0 [2026-09-22 02:58:29 TP0] Decode batch, #running-req: 1, #full token: 1694, full token usage: 0.02, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.49, #queue-req: 0 [2026-09-22 02:58:29 TP0] Decode batch, #running-req: 1, #full token: 1734, full token usage: 0.02, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.42, #queue-req: 0 [2026-09-22 02:58:29 TP0] Decode batch, #running-req: 1, #full token: 1774, full token usage: 0.02, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.58, #queue-req: 0 [2026-09-22 02:58:30 TP0] Decode batch, #running-req: 1, #full token: 1814, full token usage: 0.02, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.34, #queue-req: 0 [2026-09-22 02:58:30 TP0] Decode batch, #running-req: 1, #full token: 1854, full token usage: 0.02, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.37, #queue-req: 0 [2026-09-22 02:58:30 TP0] Decode batch, #running-req: 1, #full token: 1894, full token usage: 0.02, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.44, #queue-req: 0 [2026-09-22 02:58:31 TP0] Decode batch, #running-req: 1, #full token: 1934, full token usage: 0.02, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.26, #queue-req: 0 [2026-09-22 02:58:31 TP0] Decode batch, #running-req: 1, #full token: 1974, full token usage: 0.02, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.68, #queue-req: 0 [2026-09-22 02:58:31 TP0] Decode batch, #running-req: 1, #full token: 2014, full token usage: 0.02, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.65, #queue-req: 0 [2026-09-22 02:58:32 TP0] Decode batch, #running-req: 1, #full token: 2054, full token usage: 0.02, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.70, #queue-req: 0 [2026-09-22 02:58:32 TP0] Decode batch, #running-req: 1, #full token: 2094, full token usage: 0.02, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.79, #queue-req: 0 [2026-09-22 02:58:32 TP0] Decode batch, #running-req: 1, #full token: 2134, full token usage: 0.02, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.77, #queue-req: 0 [2026-09-22 02:58:33 TP0] Decode batch, #running-req: 1, #full token: 2174, full token usage: 0.02, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.33, #queue-req: 0 [2026-09-22 02:58:33 TP0] Decode batch, #running-req: 1, #full token: 2214, full token usage: 0.02, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.77, #queue-req: 0 [2026-09-22 02:58:33 TP0] Decode batch, #running-req: 1, #full token: 2254, full token usage: 0.02, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 126.13, #queue-req: 0 [2026-09-22 02:58:34 TP0] Decode batch, #running-req: 1, #full token: 2294, full token usage: 0.03, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.71, #queue-req: 0 [2026-09-22 02:58:34 TP0] Decode batch, #running-req: 1, #full token: 2334, full token usage: 0.03, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.32, #queue-req: 0 [2026-09-22 02:58:34 TP0] Decode batch, #running-req: 1, #full token: 2374, full token usage: 0.03, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.35, #queue-req: 0 [2026-09-22 02:58:35 TP0] Decode batch, #running-req: 1, #full token: 2414, full token usage: 0.03, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.24, #queue-req: 0 [2026-09-22 02:58:35 TP0] Decode batch, #running-req: 1, #full token: 2454, full token usage: 0.03, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.25, #queue-req: 0 [2026-09-22 02:58:35 TP0] Decode batch, #running-req: 1, #full token: 2494, full token usage: 0.03, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.35, #queue-req: 0 [2026-09-22 02:58:35 TP0] Decode batch, #running-req: 1, #full token: 2534, full token usage: 0.03, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.19, #queue-req: 0 [2026-09-22 02:58:36 TP0] Decode batch, #running-req: 1, #full token: 2574, full token usage: 0.03, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.18, #queue-req: 0 [2026-09-22 02:58:36 TP0] Decode batch, #running-req: 1, #full token: 2614, full token usage: 0.03, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.14, #queue-req: 0 [2026-09-22 02:58:36 TP0] Decode batch, #running-req: 1, #full token: 2654, full token usage: 0.03, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.57, #queue-req: 0 [2026-09-22 02:58:37 TP0] Decode batch, #running-req: 1, #full token: 2694, full token usage: 0.03, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.28, #queue-req: 0 [2026-09-22 02:58:37 TP0] Decode batch, #running-req: 1, #full token: 2734, full token usage: 0.03, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.25, #queue-req: 0 [2026-09-22 02:58:37 TP0] Decode batch, #running-req: 1, #full token: 2774, full token usage: 0.03, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.17, #queue-req: 0 [2026-09-22 02:58:38 TP0] Decode batch, #running-req: 1, #full token: 2814, full token usage: 0.03, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.40, #queue-req: 0 [2026-09-22 02:58:38 TP0] Decode batch, #running-req: 1, #full token: 2854, full token usage: 0.03, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.42, #queue-req: 0 [2026-09-22 02:58:38 TP0] Decode batch, #running-req: 1, #full token: 2894, full token usage: 0.03, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.63, #queue-req: 0 [2026-09-22 02:58:39 TP0] Decode batch, #running-req: 1, #full token: 2934, full token usage: 0.03, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.40, #queue-req: 0 [2026-09-22 02:58:39 TP0] Decode batch, #running-req: 1, #full token: 2974, full token usage: 0.03, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.82, #queue-req: 0 [2026-09-22 02:58:39 TP0] Decode batch, #running-req: 1, #full token: 3014, full token usage: 0.03, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.78, #queue-req: 0 [2026-09-22 02:58:40 TP0] Decode batch, #running-req: 1, #full token: 3054, full token usage: 0.03, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.77, #queue-req: 0 [2026-09-22 02:58:40 TP0] Decode batch, #running-req: 1, #full token: 3094, full token usage: 0.03, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.17, #queue-req: 0 [2026-09-22 02:58:40 TP0] Decode batch, #running-req: 1, #full token: 3134, full token usage: 0.03, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.15, #queue-req: 0 [2026-09-22 02:58:41 TP0] Decode batch, #running-req: 1, #full token: 3174, full token usage: 0.03, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.43, #queue-req: 0 [2026-09-22 02:58:41 TP0] Decode batch, #running-req: 1, #full token: 3214, full token usage: 0.04, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.96, #queue-req: 0 [2026-09-22 02:58:41 TP0] Decode batch, #running-req: 1, #full token: 3254, full token usage: 0.04, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.95, #queue-req: 0 [2026-09-22 02:58:42 TP0] Decode batch, #running-req: 1, #full token: 3294, full token usage: 0.04, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.64, #queue-req: 0 [2026-09-22 02:58:42 TP0] Decode batch, #running-req: 1, #full token: 3334, full token usage: 0.04, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.62, #queue-req: 0 [2026-09-22 02:58:42 TP0] Decode batch, #running-req: 1, #full token: 3374, full token usage: 0.04, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.69, #queue-req: 0 [2026-09-22 02:58:42 TP0] Decode batch, #running-req: 1, #full token: 3414, full token usage: 0.04, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 125.03, #queue-req: 0 [2026-09-22 02:58:43 TP0] Decode batch, #running-req: 1, #full token: 3454, full token usage: 0.04, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.78, #queue-req: 0 [2026-09-22 02:58:43 TP0] Decode batch, #running-req: 1, #full token: 3494, full token usage: 0.04, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.41, #queue-req: 0 [2026-09-22 02:58:43 TP0] Decode batch, #running-req: 1, #full token: 3534, full token usage: 0.04, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.29, #queue-req: 0 [2026-09-22 02:58:44 TP0] Decode batch, #running-req: 1, #full token: 3574, full token usage: 0.04, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.31, #queue-req: 0 [2026-09-22 02:58:44 TP0] Decode batch, #running-req: 1, #full token: 3614, full token usage: 0.04, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.98, #queue-req: 0 [2026-09-22 02:58:44 TP0] Decode batch, #running-req: 1, #full token: 3654, full token usage: 0.04, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.84, #queue-req: 0 [2026-09-22 02:58:45 TP0] Decode batch, #running-req: 1, #full token: 3694, full token usage: 0.04, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.87, #queue-req: 0 [2026-09-22 02:58:45 TP0] Decode batch, #running-req: 1, #full token: 3734, full token usage: 0.04, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.25, #queue-req: 0 [2026-09-22 02:58:45 TP0] Decode batch, #running-req: 1, #full token: 3774, full token usage: 0.04, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.55, #queue-req: 0 [2026-09-22 02:58:46 TP0] Decode batch, #running-req: 1, #full token: 3814, full token usage: 0.04, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.35, #queue-req: 0 [2026-09-22 02:58:46 TP0] Decode batch, #running-req: 1, #full token: 3854, full token usage: 0.04, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.31, #queue-req: 0 [2026-09-22 02:58:46 TP0] Decode batch, #running-req: 1, #full token: 3894, full token usage: 0.04, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.50, #queue-req: 0 [2026-09-22 02:58:47 TP0] Decode batch, #running-req: 1, #full token: 3934, full token usage: 0.04, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.05, #queue-req: 0 [2026-09-22 02:58:47 TP0] Decode batch, #running-req: 1, #full token: 3974, full token usage: 0.04, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.95, #queue-req: 0 [2026-09-22 02:58:47 TP0] Decode batch, #running-req: 1, #full token: 4014, full token usage: 0.04, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.34, #queue-req: 0 [2026-09-22 02:58:48 TP0] Decode batch, #running-req: 1, #full token: 4054, full token usage: 0.04, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.41, #queue-req: 0 [2026-09-22 02:58:48 TP0] Decode batch, #running-req: 1, #full token: 4094, full token usage: 0.04, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 124.51, #queue-req: 0 [2026-09-22 02:58:48 TP0] Decode batch, #running-req: 1, #full token: 4134, full token usage: 0.05, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.79, #queue-req: 0 [2026-09-22 02:58:49 TP0] Decode batch, #running-req: 1, #full token: 4174, full token usage: 0.05, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.35, #queue-req: 0 [2026-09-22 02:58:49 TP0] Decode batch, #running-req: 1, #full token: 4214, full token usage: 0.05, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.69, #queue-req: 0 [2026-09-22 02:58:49 TP0] Decode batch, #running-req: 1, #full token: 4254, full token usage: 0.05, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.75, #queue-req: 0 [2026-09-22 02:58:50 TP0] Decode batch, #running-req: 1, #full token: 4294, full token usage: 0.05, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.61, #queue-req: 0 [2026-09-22 02:58:50 TP0] Decode batch, #running-req: 1, #full token: 4334, full token usage: 0.05, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.66, #queue-req: 0 [2026-09-22 02:58:50 TP0] Decode batch, #running-req: 1, #full token: 4374, full token usage: 0.05, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.84, #queue-req: 0 [2026-09-22 02:58:51 TP0] Decode batch, #running-req: 1, #full token: 4414, full token usage: 0.05, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.69, #queue-req: 0 [2026-09-22 02:58:51 TP0] Decode batch, #running-req: 1, #full token: 4454, full token usage: 0.05, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.99, #queue-req: 0 [2026-09-22 02:58:51 TP0] Decode batch, #running-req: 1, #full token: 4494, full token usage: 0.05, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.99, #queue-req: 0 [2026-09-22 02:58:52 TP0] Decode batch, #running-req: 1, #full token: 4534, full token usage: 0.05, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.94, #queue-req: 0 [2026-09-22 02:58:52 TP0] Decode batch, #running-req: 1, #full token: 4574, full token usage: 0.05, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.73, #queue-req: 0 [2026-09-22 02:58:52 TP0] Decode batch, #running-req: 1, #full token: 4614, full token usage: 0.05, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.69, #queue-req: 0 [2026-09-22 02:58:52 TP0] Decode batch, #running-req: 1, #full token: 4654, full token usage: 0.05, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.47, #queue-req: 0 [2026-09-22 02:58:53 TP0] Decode batch, #running-req: 1, #full token: 4694, full token usage: 0.05, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.39, #queue-req: 0 [2026-09-22 02:58:53 TP0] Decode batch, #running-req: 1, #full token: 4734, full token usage: 0.05, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.13, #queue-req: 0 [2026-09-22 02:58:53 TP0] Decode batch, #running-req: 1, #full token: 4774, full token usage: 0.05, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.61, #queue-req: 0 [2026-09-22 02:58:54 TP0] Decode batch, #running-req: 1, #full token: 4814, full token usage: 0.05, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.64, #queue-req: 0 [2026-09-22 02:58:54 TP0] Decode batch, #running-req: 1, #full token: 4854, full token usage: 0.05, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.68, #queue-req: 0 [2026-09-22 02:58:54 TP0] Decode batch, #running-req: 1, #full token: 4894, full token usage: 0.05, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.35, #queue-req: 0 [2026-09-22 02:58:55 TP0] Decode batch, #running-req: 1, #full token: 4934, full token usage: 0.05, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.25, #queue-req: 0 [2026-09-22 02:58:55 TP0] Decode batch, #running-req: 1, #full token: 4974, full token usage: 0.05, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.27, #queue-req: 0 [2026-09-22 02:58:55 TP0] Decode batch, #running-req: 1, #full token: 5014, full token usage: 0.05, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.87, #queue-req: 0 [2026-09-22 02:58:56 TP0] Decode batch, #running-req: 1, #full token: 5054, full token usage: 0.06, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.47, #queue-req: 0 [2026-09-22 02:58:56 TP0] Decode batch, #running-req: 1, #full token: 5094, full token usage: 0.06, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.45, #queue-req: 0 [2026-09-22 02:58:56 TP0] Decode batch, #running-req: 1, #full token: 5134, full token usage: 0.06, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.37, #queue-req: 0 [2026-09-22 02:58:57 TP0] Decode batch, #running-req: 1, #full token: 5174, full token usage: 0.06, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.40, #queue-req: 0 [2026-09-22 02:58:57 TP0] Decode batch, #running-req: 1, #full token: 5214, full token usage: 0.06, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.71, #queue-req: 0 [2026-09-22 02:58:57 TP0] Decode batch, #running-req: 1, #full token: 5254, full token usage: 0.06, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.34, #queue-req: 0 [2026-09-22 02:58:58 TP0] Decode batch, #running-req: 1, #full token: 5294, full token usage: 0.06, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.96, #queue-req: 0 [2026-09-22 02:58:58 TP0] Decode batch, #running-req: 1, #full token: 5334, full token usage: 0.06, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.06, #queue-req: 0 [2026-09-22 02:58:58 TP0] Decode batch, #running-req: 1, #full token: 5374, full token usage: 0.06, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.09, #queue-req: 0 [2026-09-22 02:58:59 TP0] Decode batch, #running-req: 1, #full token: 5414, full token usage: 0.06, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.12, #queue-req: 0 [2026-09-22 02:58:59 TP0] Decode batch, #running-req: 1, #full token: 5454, full token usage: 0.06, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.79, #queue-req: 0 [2026-09-22 02:58:59 TP0] Decode batch, #running-req: 1, #full token: 5494, full token usage: 0.06, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.00, #queue-req: 0 [2026-09-22 02:59:00 TP0] Decode batch, #running-req: 1, #full token: 5534, full token usage: 0.06, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.95, #queue-req: 0 [2026-09-22 02:59:00 TP0] Decode batch, #running-req: 1, #full token: 5574, full token usage: 0.06, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.96, #queue-req: 0 [2026-09-22 02:59:00 TP0] Decode batch, #running-req: 1, #full token: 5614, full token usage: 0.06, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.25, #queue-req: 0 [2026-09-22 02:59:01 TP0] Decode batch, #running-req: 1, #full token: 5654, full token usage: 0.06, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 122.13, #queue-req: 0 [2026-09-22 02:59:01 TP0] Decode batch, #running-req: 1, #full token: 5694, full token usage: 0.06, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.65, #queue-req: 0 [2026-09-22 02:59:01 TP0] Decode batch, #running-req: 1, #full token: 5734, full token usage: 0.06, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.96, #queue-req: 0 [2026-09-22 02:59:02 TP0] Decode batch, #running-req: 1, #full token: 5774, full token usage: 0.06, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.83, #queue-req: 0 [2026-09-22 02:59:02 TP0] Decode batch, #running-req: 1, #full token: 5814, full token usage: 0.06, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.83, #queue-req: 0 [2026-09-22 02:59:02 TP0] Decode batch, #running-req: 1, #full token: 5854, full token usage: 0.06, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.77, #queue-req: 0 [2026-09-22 02:59:03 TP0] Decode batch, #running-req: 1, #full token: 5894, full token usage: 0.06, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.69, #queue-req: 0 [2026-09-22 02:59:03 TP0] Decode batch, #running-req: 1, #full token: 5934, full token usage: 0.06, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.39, #queue-req: 0 [2026-09-22 02:59:03 TP0] Decode batch, #running-req: 1, #full token: 5974, full token usage: 0.07, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.52, #queue-req: 0 [2026-09-22 02:59:04 TP0] Decode batch, #running-req: 1, #full token: 6014, full token usage: 0.07, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.30, #queue-req: 0 [2026-09-22 02:59:04 TP0] Decode batch, #running-req: 1, #full token: 6054, full token usage: 0.07, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.46, #queue-req: 0 [2026-09-22 02:59:04 TP0] Decode batch, #running-req: 1, #full token: 6094, full token usage: 0.07, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.89, #queue-req: 0 [2026-09-22 02:59:05 TP0] Decode batch, #running-req: 1, #full token: 6134, full token usage: 0.07, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.97, #queue-req: 0 [2026-09-22 02:59:05 TP0] Decode batch, #running-req: 1, #full token: 6174, full token usage: 0.07, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.17, #queue-req: 0 [2026-09-22 02:59:05 TP0] Decode batch, #running-req: 1, #full token: 6214, full token usage: 0.07, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.25, #queue-req: 0 [2026-09-22 02:59:06 TP0] Decode batch, #running-req: 1, #full token: 6254, full token usage: 0.07, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.00, #queue-req: 0 [2026-09-22 02:59:06 TP0] Decode batch, #running-req: 1, #full token: 6294, full token usage: 0.07, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.99, #queue-req: 0 [2026-09-22 02:59:06 TP0] Decode batch, #running-req: 1, #full token: 6334, full token usage: 0.07, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.66, #queue-req: 0 [2026-09-22 02:59:07 TP0] Decode batch, #running-req: 1, #full token: 6374, full token usage: 0.07, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.60, #queue-req: 0 [2026-09-22 02:59:07 TP0] Decode batch, #running-req: 1, #full token: 6414, full token usage: 0.07, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.25, #queue-req: 0 [2026-09-22 02:59:07 TP0] Decode batch, #running-req: 1, #full token: 6454, full token usage: 0.07, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.71, #queue-req: 0 [2026-09-22 02:59:08 TP0] Decode batch, #running-req: 1, #full token: 6494, full token usage: 0.07, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.88, #queue-req: 0 [2026-09-22 02:59:08 TP0] Decode batch, #running-req: 1, #full token: 6534, full token usage: 0.07, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.61, #queue-req: 0 [2026-09-22 02:59:08 TP0] Decode batch, #running-req: 1, #full token: 6574, full token usage: 0.07, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.85, #queue-req: 0 [2026-09-22 02:59:09 TP0] Decode batch, #running-req: 1, #full token: 6614, full token usage: 0.07, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.49, #queue-req: 0 [2026-09-22 02:59:09 TP0] Decode batch, #running-req: 1, #full token: 6654, full token usage: 0.07, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.21, #queue-req: 0 [2026-09-22 02:59:09 TP0] Decode batch, #running-req: 1, #full token: 6694, full token usage: 0.07, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.06, #queue-req: 0 [2026-09-22 02:59:10 TP0] Decode batch, #running-req: 1, #full token: 6734, full token usage: 0.07, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.02, #queue-req: 0 [2026-09-22 02:59:10 TP0] Decode batch, #running-req: 1, #full token: 6774, full token usage: 0.07, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.70, #queue-req: 0 [2026-09-22 02:59:10 TP0] Decode batch, #running-req: 1, #full token: 6814, full token usage: 0.07, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.80, #queue-req: 0 [2026-09-22 02:59:11 TP0] Decode batch, #running-req: 1, #full token: 6854, full token usage: 0.08, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.05, #queue-req: 0 [2026-09-22 02:59:11 TP0] Decode batch, #running-req: 1, #full token: 6894, full token usage: 0.08, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.81, #queue-req: 0 [2026-09-22 02:59:11 TP0] Decode batch, #running-req: 1, #full token: 6934, full token usage: 0.08, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.56, #queue-req: 0 [2026-09-22 02:59:12 TP0] Decode batch, #running-req: 1, #full token: 6974, full token usage: 0.08, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.92, #queue-req: 0 [2026-09-22 02:59:12 TP0] Decode batch, #running-req: 1, #full token: 7014, full token usage: 0.08, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.16, #queue-req: 0 [2026-09-22 02:59:12 TP0] Decode batch, #running-req: 1, #full token: 7054, full token usage: 0.08, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.40, #queue-req: 0 [2026-09-22 02:59:12 TP0] Decode batch, #running-req: 1, #full token: 7094, full token usage: 0.08, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.08, #queue-req: 0 [2026-09-22 02:59:13 TP0] Decode batch, #running-req: 1, #full token: 7134, full token usage: 0.08, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.70, #queue-req: 0 [2026-09-22 02:59:13 TP0] Decode batch, #running-req: 1, #full token: 7174, full token usage: 0.08, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.55, #queue-req: 0 [2026-09-22 02:59:13 TP0] Decode batch, #running-req: 1, #full token: 7214, full token usage: 0.08, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.30, #queue-req: 0 [2026-09-22 02:59:14 TP0] Decode batch, #running-req: 1, #full token: 7254, full token usage: 0.08, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.57, #queue-req: 0 [2026-09-22 02:59:14 TP0] Decode batch, #running-req: 1, #full token: 7294, full token usage: 0.08, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.07, #queue-req: 0 [2026-09-22 02:59:14 TP0] Decode batch, #running-req: 1, #full token: 7334, full token usage: 0.08, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.69, #queue-req: 0 [2026-09-22 02:59:15 TP0] Decode batch, #running-req: 1, #full token: 7374, full token usage: 0.08, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.43, #queue-req: 0 [2026-09-22 02:59:15 TP0] Decode batch, #running-req: 1, #full token: 7414, full token usage: 0.08, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 121.12, #queue-req: 0 [2026-09-22 02:59:15 TP0] Decode batch, #running-req: 1, #full token: 7454, full token usage: 0.08, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.42, #queue-req: 0 [2026-09-22 02:59:16 TP0] Decode batch, #running-req: 1, #full token: 7494, full token usage: 0.08, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.59, #queue-req: 0 [2026-09-22 02:59:16 TP0] Decode batch, #running-req: 1, #full token: 7534, full token usage: 0.08, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.59, #queue-req: 0 [2026-09-22 02:59:16 TP0] Decode batch, #running-req: 1, #full token: 7574, full token usage: 0.08, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.22, #queue-req: 0 [2026-09-22 02:59:17 TP0] Decode batch, #running-req: 1, #full token: 7614, full token usage: 0.08, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.77, #queue-req: 0 [2026-09-22 02:59:17 TP0] Decode batch, #running-req: 1, #full token: 7654, full token usage: 0.08, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.22, #queue-req: 0 [2026-09-22 02:59:17 TP0] Decode batch, #running-req: 1, #full token: 7694, full token usage: 0.08, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.89, #queue-req: 0 [2026-09-22 02:59:18 TP0] Decode batch, #running-req: 1, #full token: 7734, full token usage: 0.08, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.16, #queue-req: 0 [2026-09-22 02:59:18 TP0] Decode batch, #running-req: 1, #full token: 7774, full token usage: 0.09, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.17, #queue-req: 0 [2026-09-22 02:59:18 TP0] Decode batch, #running-req: 1, #full token: 7814, full token usage: 0.09, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.84, #queue-req: 0 [2026-09-22 02:59:19 TP0] Decode batch, #running-req: 1, #full token: 7854, full token usage: 0.09, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.59, #queue-req: 0 [2026-09-22 02:59:19 TP0] Decode batch, #running-req: 1, #full token: 7894, full token usage: 0.09, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.97, #queue-req: 0 [2026-09-22 02:59:19 TP0] Decode batch, #running-req: 1, #full token: 7934, full token usage: 0.09, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.25, #queue-req: 0 [2026-09-22 02:59:20 TP0] Decode batch, #running-req: 1, #full token: 7974, full token usage: 0.09, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.74, #queue-req: 0 [2026-09-22 02:59:20 TP0] Decode batch, #running-req: 1, #full token: 8014, full token usage: 0.09, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.85, #queue-req: 0 [2026-09-22 02:59:20 TP0] Decode batch, #running-req: 1, #full token: 8054, full token usage: 0.09, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.67, #queue-req: 0 [2026-09-22 02:59:21 TP0] Decode batch, #running-req: 1, #full token: 8094, full token usage: 0.09, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.36, #queue-req: 0 [2026-09-22 02:59:21 TP0] Decode batch, #running-req: 1, #full token: 8134, full token usage: 0.09, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.25, #queue-req: 0 [2026-09-22 02:59:21 TP0] Decode batch, #running-req: 1, #full token: 8174, full token usage: 0.09, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 120.16, #queue-req: 0 [2026-09-22 02:59:22 TP0] Decode batch, #running-req: 1, #full token: 8214, full token usage: 0.09, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.99, #queue-req: 0 [2026-09-22 02:59:22 TP0] Decode batch, #running-req: 1, #full token: 8254, full token usage: 0.09, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.94, #queue-req: 0 [2026-09-22 02:59:22 TP0] Decode batch, #running-req: 1, #full token: 8294, full token usage: 0.09, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.92, #queue-req: 0 [2026-09-22 02:59:23 TP0] Decode batch, #running-req: 1, #full token: 8334, full token usage: 0.09, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.78, #queue-req: 0 [2026-09-22 02:59:23 TP0] Decode batch, #running-req: 1, #full token: 8374, full token usage: 0.09, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.88, #queue-req: 0 [2026-09-22 02:59:23 TP0] Decode batch, #running-req: 1, #full token: 8414, full token usage: 0.09, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.96, #queue-req: 0 [2026-09-22 02:59:24 TP0] Decode batch, #running-req: 1, #full token: 8454, full token usage: 0.09, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.78, #queue-req: 0 [2026-09-22 02:59:24 TP0] Decode batch, #running-req: 1, #full token: 8494, full token usage: 0.09, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.54, #queue-req: 0 [2026-09-22 02:59:24 TP0] Decode batch, #running-req: 1, #full token: 8534, full token usage: 0.09, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.58, #queue-req: 0 [2026-09-22 02:59:25 TP0] Decode batch, #running-req: 1, #full token: 8574, full token usage: 0.09, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.21, #queue-req: 0 [2026-09-22 02:59:25 TP0] Decode batch, #running-req: 1, #full token: 8614, full token usage: 0.09, #swa token: 167, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.77, #queue-req: 0 [2026-09-22 02:59:25 TP0] Decode batch, #running-req: 1, #full token: 8654, full token usage: 0.09, #swa token: 207, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.62, #queue-req: 0 [2026-09-22 02:59:26 TP0] Decode batch, #running-req: 1, #full token: 8694, full token usage: 0.10, #swa token: 247, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.65, #queue-req: 0 [2026-09-22 02:59:26 TP0] Decode batch, #running-req: 1, #full token: 8734, full token usage: 0.10, #swa token: 159, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.29, #queue-req: 0 [2026-09-22 02:59:26 TP0] Decode batch, #running-req: 1, #full token: 8774, full token usage: 0.10, #swa token: 199, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.78, #queue-req: 0 [2026-09-22 02:59:27 TP0] Decode batch, #running-req: 1, #full token: 8814, full token usage: 0.10, #swa token: 239, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.92, #queue-req: 0 [2026-09-22 02:59:27 TP0] Decode batch, #running-req: 1, #full token: 8854, full token usage: 0.10, #swa token: 279, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.25, #queue-req: 0 [2026-09-22 02:59:28 TP0] Decode batch, #running-req: 1, #full token: 8894, full token usage: 0.10, #swa token: 191, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.28, #queue-req: 0 [2026-09-22 02:59:28 TP0] Decode batch, #running-req: 1, #full token: 8934, full token usage: 0.10, #swa token: 231, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.21, #queue-req: 0 [2026-09-22 02:59:28 TP0] Decode batch, #running-req: 1, #full token: 8974, full token usage: 0.10, #swa token: 271, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.16, #queue-req: 0 [2026-09-22 02:59:29 TP0] Decode batch, #running-req: 1, #full token: 9014, full token usage: 0.10, #swa token: 183, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.99, #queue-req: 0 [2026-09-22 02:59:29 TP0] Decode batch, #running-req: 1, #full token: 9054, full token usage: 0.10, #swa token: 223, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.58, #queue-req: 0 [2026-09-22 02:59:29 TP0] Decode batch, #running-req: 1, #full token: 9094, full token usage: 0.10, #swa token: 263, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.84, #queue-req: 0 [2026-09-22 02:59:30 TP0] Decode batch, #running-req: 1, #full token: 9134, full token usage: 0.10, #swa token: 175, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 118.97, #queue-req: 0 [2026-09-22 02:59:30 TP0] Decode batch, #running-req: 1, #full token: 9174, full token usage: 0.10, #swa token: 215, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.11, #queue-req: 0 [2026-09-22 02:59:30 TP0] Decode batch, #running-req: 1, #full token: 9214, full token usage: 0.10, #swa token: 255, swa token usage: 0.00, cuda graph: True, gen throughput (token/s): 119.08, #queue-req: 0