Two training runs show numerical divergence. Diagnose it from the supplied evidence. Launch commands are not supplied. Do not assume the runs have identical configurations or that there is only one difference. --- Recorded environment (both runs) --- { "python": "3.11.2", "torch": "2.9.1+cu130", "cuda_build": "13.0", "driver": "580.126.20", "deepspeed": "0.14.2", "accelerate": "0.29.3", "numpy": "1.26.4", "gpu": "2 x NVIDIA GB200" } Dependency Git commits were not recorded; DeepSpeed reports git-hash=unknown. Package versions identify the environment, not a verified source location. --- Runtime observations --- Standard logs with normalized paths and neutral run labels, in original order within each stream. stdout/stderr were captured separately; their interleaving is unknown. --- run1 / stdout --- The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! [2026-10-19 21:19:00,225] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [2026-10-19 21:19:00,331] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.9  [WARNING]  using untested triton version (3.5.1), only 1.0.0 is known to be compatible  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.9  [WARNING]  using untested triton version (3.5.1), only 1.0.0 is known to be compatible [2026-10-19 21:19:01,317] [INFO] [comm.py:637:init_distributed] cdb=None [2026-10-19 21:19:01,409] [INFO] [comm.py:637:init_distributed] cdb=None [2026-10-19 21:19:01,409] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2026-10-19 21:19:02,537] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.14.2, git-hash=unknown, git-branch=unknown NCCL version 2.27.7+cuda13.0 [2026-10-19 21:19:03,150] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2026-10-19 21:19:03,151] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2026-10-19 21:19:03,151] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2026-10-19 21:19:03,151] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2026-10-19 21:19:03,151] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=Adam type= [2026-10-19 21:19:03,151] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.float16 ZeRO stage 1 optimizer [2026-10-19 21:19:03,151] [INFO] [stage_1_and_2.py:194:__init__] Reduce bucket size 1048576 [2026-10-19 21:19:03,151] [INFO] [stage_1_and_2.py:195:__init__] Allgather bucket size 100000000 [2026-10-19 21:19:03,151] [INFO] [stage_1_and_2.py:196:__init__] CPU Offload: False [2026-10-19 21:19:03,151] [INFO] [stage_1_and_2.py:197:__init__] Round robin gradient partitioning: False [2026-10-19 21:19:03,279] [INFO] [utils.py:779:see_memory_usage] Before initializing optimizer states [2026-10-19 21:19:03,280] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.08 GB CA 0.08 GB Max_CA 0 GB [2026-10-19 21:19:03,280] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 263.66 GB, percent = 27.7% [2026-10-19 21:19:03,367] [INFO] [utils.py:779:see_memory_usage] After initializing optimizer states [2026-10-19 21:19:03,367] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.09 GB CA 0.12 GB Max_CA 0 GB [2026-10-19 21:19:03,368] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 263.78 GB, percent = 27.7% [2026-10-19 21:19:03,368] [INFO] [stage_1_and_2.py:589:__init__] optimizer state initialized [2026-10-19 21:19:03,443] [INFO] [utils.py:779:see_memory_usage] After initializing ZeRO optimizer [2026-10-19 21:19:03,444] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.06 GB CA 0.12 GB Max_CA 0 GB [2026-10-19 21:19:03,444] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 263.83 GB, percent = 27.7% [2026-10-19 21:19:03,454] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2026-10-19 21:19:03,455] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler [2026-10-19 21:19:03,455] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2026-10-19 21:19:03,455] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.001], mom=[(0.9, 0.999)] [2026-10-19 21:19:03,455] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2026-10-19 21:19:03,455] [INFO] [config.py:1000:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2026-10-19 21:19:03,455] [INFO] [config.py:1000:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2026-10-19 21:19:03,455] [INFO] [config.py:1000:print] amp_enabled .................. False [2026-10-19 21:19:03,455] [INFO] [config.py:1000:print] amp_params ................... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] bfloat16_enabled ............. False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] bfloat16_immediate_grad_update False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] checkpoint_parallel_write_pipeline False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] checkpoint_tag_validation_enabled True [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] checkpoint_tag_validation_fail False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] comms_config ................. [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] communication_data_type ...... None [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] compile_config ............... enabled=False backend='inductor' kwargs={} [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] curriculum_enabled_legacy .... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] curriculum_params_legacy ..... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] data_efficiency_enabled ...... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] dataloader_drop_last ......... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] disable_allgather ............ False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] dump_state ................... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] dynamic_loss_scale_args ...... None [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_enabled ........... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_gas_boundary_resolution 1 [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_layer_name ........ bert.encoder.layer [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_layer_num ......... 0 [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_max_iter .......... 100 [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_stability ......... 1e-06 [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_tol ............... 0.01 [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] eigenvalue_verbose ........... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] elasticity_enabled ........... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] fp16_auto_cast ............... False [2026-10-19 21:19:03,456] [INFO] [config.py:1000:print] fp16_enabled ................. True [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] fp16_master_weights_and_gradients False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] global_rank .................. 0 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] grad_accum_dtype ............. None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] gradient_accumulation_steps .. 1 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] gradient_clipping ............ 1.0 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] gradient_predivide_factor .... 1.0 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] graph_harvesting ............. False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] initial_dynamic_scale ........ 65536 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] load_universal_checkpoint .... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] loss_scale ................... 0 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] memory_breakdown ............. False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] mics_hierarchial_params_gather False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] mics_shard_size .............. -1 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] optimizer_legacy_fusion ...... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] optimizer_name ............... None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] optimizer_params ............. None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] pld_enabled .................. False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] pld_params ................... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] prescale_gradients ........... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] scheduler_name ............... None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] scheduler_params ............. None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] seq_parallel_communication_data_type torch.float32 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] sparse_attention ............. None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] sparse_gradients_enabled ..... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] steps_per_print .............. inf [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] train_batch_size ............. 12 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] train_micro_batch_size_per_gpu 6 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] use_data_before_expert_parallel_ False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] use_node_local_storage ....... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] wall_clock_breakdown ......... False [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] weight_quantization_config ... None [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] world_size ................... 2 [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] zero_allow_untested_optimizer True [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1048576 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=100000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=2000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2026-10-19 21:19:03,457] [INFO] [config.py:1000:print] zero_enabled ................. True [2026-10-19 21:19:03,458] [INFO] [config.py:1000:print] zero_force_ds_cpu_optimizer .. True [2026-10-19 21:19:03,458] [INFO] [config.py:1000:print] zero_optimization_stage ...... 1 [2026-10-19 21:19:03,458] [INFO] [config.py:986:print_user_config] json = { "train_micro_batch_size_per_gpu": 6, "steps_per_print": inf, "prescale_gradients": false, "zero_allow_untested_optimizer": true, "gradient_accumulation_steps": 1, "bf16": { "enabled": false }, "fp16": { "enabled": true }, "wall_clock_breakdown": false, "gradient_clipping": 1.0, "zero_optimization": { "stage": 1, "allgather_partitions": true, "reduce_scatter": true, "allgather_bucket_size": 1.000000e+08, "reduce_bucket_size": 1.048576e+06, "stage3_max_reuse_distance": 2.000000e+09, "overlap_comm": false, "contiguous_gradients": true } } [2026-10-19 21:19:03,582] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 4294967296, reducing to 2147483648 rank [0] global_step [0] loss [0.139528513] grad_norm [0.0] [2026-10-19 21:19:03,621] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 2147483648, reducing to 1073741824 rank [0] global_step [1] loss [0.00953674316] grad_norm [0.0] [2026-10-19 21:19:03,660] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1073741824, reducing to 536870912 rank [0] global_step [2] loss [0.000167429447] grad_norm [0.0] [2026-10-19 21:19:03,697] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 536870912, reducing to 268435456 rank [0] global_step [3] loss [0.017284289] grad_norm [0.0] [2026-10-19 21:19:03,735] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 268435456, reducing to 134217728 rank [0] global_step [4] loss [0.0800657868] grad_norm [0.0] [2026-10-19 21:19:03,773] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 134217728, reducing to 67108864 rank [0] global_step [5] loss [0.0393968225] grad_norm [0.0] [2026-10-19 21:19:03,814] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 67108864, reducing to 33554432 rank [0] global_step [6] loss [0.026358664] grad_norm [0.0] [2026-10-19 21:19:03,856] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 33554432, reducing to 16777216 rank [0] global_step [7] loss [0.232495368] grad_norm [0.0] [2026-10-19 21:19:03,964] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 16777216, reducing to 8388608 rank [0] global_step [8] loss [0.680947304] grad_norm [0.0] [2026-10-19 21:19:04,066] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 8388608, reducing to 4194304 rank [0] global_step [9] loss [0.125664711] grad_norm [0.0] [2026-10-19 21:19:04,170] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 4194304, reducing to 2097152 rank [0] global_step [10] loss [0.00123596191] grad_norm [0.0] [2026-10-19 21:19:04,276] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 2097152, reducing to 1048576 rank [0] global_step [11] loss [0.386364222] grad_norm [0.0] [2026-10-19 21:19:04,381] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1048576, reducing to 524288 rank [0] global_step [12] loss [0.263357401] grad_norm [0.0] [2026-10-19 21:19:04,539] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 524288, reducing to 262144 rank [0] global_step [13] loss [0.314075947] grad_norm [0.0] [2026-10-19 21:19:04,715] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 131072 rank [0] global_step [14] loss [0.0901025683] grad_norm [0.0] [2026-10-19 21:19:04,887] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072, reducing to 65536 rank [0] global_step [15] loss [0.00583940744] grad_norm [0.0] rank [0] global_step [16] loss [0.0467894077] grad_norm [0.00197029114] rank [0] global_step [17] loss [0.00145053864] grad_norm [0.00138282776] rank [0] global_step [18] loss [0.219383374] grad_norm [0.000346183777] rank [0] global_step [19] loss [0.12809968] grad_norm [0.00147438049] --- run1 / stderr --- /opt/opguard/.venv_opguard/lib/python3.11/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml warnings.warn( /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run1/deps/deepspeed/runtime/zero/linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. @autocast_custom_fwd /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run1/deps/deepspeed/runtime/zero/linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead. @autocast_custom_bwd /opt/opguard/.venv_opguard/lib/python3.11/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml warnings.warn( /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run1/deps/deepspeed/runtime/zero/linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. @autocast_custom_fwd /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run1/deps/deepspeed/runtime/zero/linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead. @autocast_custom_bwd [W1019 21:19:01.708631404 ProcessGroupNCCL.cpp:924] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [W1019 21:19:01.800671515 ProcessGroupNCCL.cpp:924] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [OpGuard]: meta_vars 1-154168-154168 init (size=2)[OpGuard]: meta_vars 0-154167-154167 init (size=2) [INFO] Execution traces: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/baseline-trace-1/output/exec_trace, Source files: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/baseline-trace-1/output/exec_trace/source_files[INFO] Execution traces: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/baseline-trace-1/output/exec_trace, Source files: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/baseline-trace-1/output/exec_trace/source_files /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end). return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end). return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. return original_fn(args[0], *args[1:], **kwargs) [OpGuard]: trace buffer 1-154168-154168 force dump all traces and tensors [OpGuard]: trace buffer 0-154167-154167 force dump all traces and tensors [rank1]:[W1019 21:19:07.958377069 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) [rank0]:[W1019 21:19:07.236250529 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) [rank0]:[W1019 21:19:08.438463548 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) [W1019 21:19:09.602545498 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) --- run2 / stdout --- The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! [2026-10-19 21:19:35,663] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-10-19 21:19:35,697] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.9  [WARNING]  using untested triton version (3.5.1), only 1.0.0 is known to be compatible  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.9  [WARNING]  using untested triton version (3.5.1), only 1.0.0 is known to be compatible [2026-10-19 21:19:36,696] [INFO] [comm.py:637:init_distributed] cdb=None [2026-10-19 21:19:36,696] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2026-10-19 21:19:36,704] [INFO] [comm.py:637:init_distributed] cdb=None [2026-10-19 21:19:37,870] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.14.2, git-hash=unknown, git-branch=unknown NCCL version 2.27.7+cuda13.0 [2026-10-19 21:19:38,449] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2026-10-19 21:19:38,450] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2026-10-19 21:19:38,450] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2026-10-19 21:19:38,450] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2026-10-19 21:19:38,450] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=Adam type= [2026-10-19 21:19:38,450] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.float16 ZeRO stage 1 optimizer [2026-10-19 21:19:38,450] [INFO] [stage_1_and_2.py:196:__init__] Reduce bucket size 1048576 [2026-10-19 21:19:38,450] [INFO] [stage_1_and_2.py:197:__init__] Allgather bucket size 100000000 [2026-10-19 21:19:38,450] [INFO] [stage_1_and_2.py:198:__init__] CPU Offload: False [2026-10-19 21:19:38,450] [INFO] [stage_1_and_2.py:199:__init__] Round robin gradient partitioning: False [2026-10-19 21:19:38,593] [INFO] [utils.py:779:see_memory_usage] Before initializing optimizer states [2026-10-19 21:19:38,594] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.08 GB CA 0.08 GB Max_CA 0 GB [2026-10-19 21:19:38,594] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 263.96 GB, percent = 27.7% [2026-10-19 21:19:38,676] [INFO] [utils.py:779:see_memory_usage] After initializing optimizer states [2026-10-19 21:19:38,677] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.09 GB CA 0.12 GB Max_CA 0 GB [2026-10-19 21:19:38,677] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 264.0 GB, percent = 27.7% [2026-10-19 21:19:38,677] [INFO] [stage_1_and_2.py:591:__init__] optimizer state initialized [2026-10-19 21:19:38,745] [INFO] [utils.py:779:see_memory_usage] After initializing ZeRO optimizer [2026-10-19 21:19:38,745] [INFO] [utils.py:780:see_memory_usage] MA 0.06 GB Max_MA 0.06 GB CA 0.12 GB Max_CA 0 GB [2026-10-19 21:19:38,745] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory: used = 264.02 GB, percent = 27.7% [2026-10-19 21:19:38,755] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2026-10-19 21:19:38,755] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler [2026-10-19 21:19:38,755] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2026-10-19 21:19:38,755] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.001], mom=[(0.9, 0.999)] [2026-10-19 21:19:38,755] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] amp_enabled .................. False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] amp_params ................... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] bfloat16_enabled ............. False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] bfloat16_immediate_grad_update False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] checkpoint_parallel_write_pipeline False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] checkpoint_tag_validation_enabled True [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] checkpoint_tag_validation_fail False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] comms_config ................. [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] communication_data_type ...... None [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] compile_config ............... enabled=False backend='inductor' kwargs={} [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] curriculum_enabled_legacy .... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] curriculum_params_legacy ..... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] data_efficiency_enabled ...... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] dataloader_drop_last ......... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] disable_allgather ............ False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] dump_state ................... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] dynamic_loss_scale_args ...... None [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] eigenvalue_enabled ........... False [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] eigenvalue_gas_boundary_resolution 1 [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] eigenvalue_layer_name ........ bert.encoder.layer [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] eigenvalue_layer_num ......... 0 [2026-10-19 21:19:38,756] [INFO] [config.py:1000:print] eigenvalue_max_iter .......... 100 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] eigenvalue_stability ......... 1e-06 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] eigenvalue_tol ............... 0.01 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] eigenvalue_verbose ........... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] elasticity_enabled ........... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] fp16_auto_cast ............... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] fp16_enabled ................. True [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] fp16_master_weights_and_gradients False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] global_rank .................. 0 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] grad_accum_dtype ............. None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] gradient_accumulation_steps .. 1 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] gradient_clipping ............ 1.0 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] gradient_predivide_factor .... 1.0 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] graph_harvesting ............. False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] initial_dynamic_scale ........ 65536 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] load_universal_checkpoint .... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] loss_scale ................... 0 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] memory_breakdown ............. False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] mics_hierarchial_params_gather False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] mics_shard_size .............. -1 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] optimizer_legacy_fusion ...... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] optimizer_name ............... None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] optimizer_params ............. None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] pld_enabled .................. False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] pld_params ................... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] prescale_gradients ........... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] scheduler_name ............... None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] scheduler_params ............. None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] seq_parallel_communication_data_type torch.float32 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] sparse_attention ............. None [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] sparse_gradients_enabled ..... False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] steps_per_print .............. inf [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] train_batch_size ............. 12 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] train_micro_batch_size_per_gpu 6 [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] use_data_before_expert_parallel_ False [2026-10-19 21:19:38,757] [INFO] [config.py:1000:print] use_node_local_storage ....... False [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] wall_clock_breakdown ......... False [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] weight_quantization_config ... None [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] world_size ................... 2 [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] zero_allow_untested_optimizer True [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1048576 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=100000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=2000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] zero_enabled ................. True [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] zero_force_ds_cpu_optimizer .. True [2026-10-19 21:19:38,758] [INFO] [config.py:1000:print] zero_optimization_stage ...... 1 [2026-10-19 21:19:38,758] [INFO] [config.py:986:print_user_config] json = { "train_micro_batch_size_per_gpu": 6, "steps_per_print": inf, "prescale_gradients": false, "zero_allow_untested_optimizer": true, "gradient_accumulation_steps": 1, "bf16": { "enabled": false }, "fp16": { "enabled": true }, "wall_clock_breakdown": false, "gradient_clipping": 1.0, "zero_optimization": { "stage": 1, "allgather_partitions": true, "reduce_scatter": true, "allgather_bucket_size": 1.000000e+08, "reduce_bucket_size": 1.048576e+06, "stage3_max_reuse_distance": 2.000000e+09, "overlap_comm": true, "contiguous_gradients": true } } [2026-10-19 21:19:38,881] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 4294967296, reducing to 2147483648 rank [0] global_step [0] loss [0.139528513] grad_norm [0.0] [2026-10-19 21:19:38,923] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 2147483648, reducing to 1073741824 rank [0] global_step [1] loss [0.00953674316] grad_norm [0.0] [2026-10-19 21:19:38,962] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1073741824, reducing to 536870912 rank [0] global_step [2] loss [0.000167429447] grad_norm [0.0] [2026-10-19 21:19:39,001] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 536870912, reducing to 268435456 rank [0] global_step [3] loss [0.017284289] grad_norm [0.0] [2026-10-19 21:19:39,039] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 268435456, reducing to 134217728 rank [0] global_step [4] loss [0.0800657868] grad_norm [0.0] [2026-10-19 21:19:39,077] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 134217728, reducing to 67108864 rank [0] global_step [5] loss [0.0393968225] grad_norm [0.0] [2026-10-19 21:19:39,117] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 67108864, reducing to 33554432 rank [0] global_step [6] loss [0.026358664] grad_norm [0.0] [2026-10-19 21:19:39,192] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 33554432, reducing to 16777216 rank [0] global_step [7] loss [0.232495368] grad_norm [0.0] [2026-10-19 21:19:39,286] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 16777216, reducing to 8388608 rank [0] global_step [8] loss [0.680947304] grad_norm [0.0] [2026-10-19 21:19:39,376] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 8388608, reducing to 4194304 rank [0] global_step [9] loss [0.125664711] grad_norm [0.0] [2026-10-19 21:19:39,465] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 4194304, reducing to 2097152 rank [0] global_step [10] loss [0.00123596191] grad_norm [0.0] [2026-10-19 21:19:39,555] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 2097152, reducing to 1048576 rank [0] global_step [11] loss [0.386364222] grad_norm [0.0] [2026-10-19 21:19:39,678] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1048576, reducing to 524288 rank [0] global_step [12] loss [0.263357401] grad_norm [0.0] [2026-10-19 21:19:39,804] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 524288, reducing to 262144 rank [0] global_step [13] loss [0.314075947] grad_norm [0.0] [2026-10-19 21:19:39,932] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 131072 rank [0] global_step [14] loss [0.0901025683] grad_norm [0.0] [2026-10-19 21:19:40,070] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072, reducing to 65536 rank [0] global_step [15] loss [0.00583940744] grad_norm [0.0] rank [0] global_step [16] loss [0.0467894077] grad_norm [0.00204745238] rank [0] global_step [17] loss [0.00145053864] grad_norm [0.00151081756] rank [0] global_step [18] loss [0.219383374] grad_norm [0.00111055549] rank [0] global_step [19] loss [0.129150391] grad_norm [0.00158956705] --- run2 / stderr --- /opt/opguard/.venv_opguard/lib/python3.11/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml warnings.warn( /opt/opguard/.venv_opguard/lib/python3.11/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml warnings.warn( /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run2/deps/deepspeed/runtime/zero/linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. @autocast_custom_fwd /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run2/deps/deepspeed/runtime/zero/linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead. @autocast_custom_bwd /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run2/deps/deepspeed/runtime/zero/linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. @autocast_custom_fwd /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/workloads/run2/deps/deepspeed/runtime/zero/linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead. @autocast_custom_bwd [W1019 21:19:36.087727907 ProcessGroupNCCL.cpp:924] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [W1019 21:19:36.095211100 ProcessGroupNCCL.cpp:924] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [OpGuard]: meta_vars 0-154589-154589 init (size=2)[OpGuard]: meta_vars 1-154590-154590 init (size=2) [INFO] Execution traces: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/candidate-trace-1/output/exec_trace, Source files: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/candidate-trace-1/output/exec_trace/source_files[INFO] Execution traces: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/candidate-trace-1/output/exec_trace, Source files: /opt/opguard/OpGuard-runs/demo-case5-paired-20261019/experiments/5dc7374b7a5949caae53ea82da8ee78d/candidate-trace-1/output/exec_trace/source_files /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end). return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end). return original_fn(args[0], *args[1:], **kwargs) /opt/opguard/OpGuard/opguard/wrapped_ops.py:37: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size. return original_fn(args[0], *args[1:], **kwargs) [OpGuard]: trace buffer 1-154590-154590 force dump all traces and tensors [OpGuard]: trace buffer 0-154589-154589 force dump all traces and tensors [rank1]:[W1019 21:19:42.583687854 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) [rank0]:[W1019 21:19:42.874193324 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) [rank0]:[W1019 21:19:42.148923702 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) [W1019 21:19:43.121688478 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())