OpGuard

Debug training differences
with your AI agent.

Give your coding agent two training runs. OpGuard finds where their computations diverge and provides the tensor comparisons, call stacks and source your agent needs to investigate why.

Get started Try the diagnosis demo

Works with coding agents that can run shell commands · Open source · Apache-2.0

5 min vs. 5 days to localize

20+ production issues diagnosed

Source + trace evidence your agent can inspect

OSDI ’26 Best Paper Award Nominee

Get started

Two runs. One prompt.

Install OpGuard where you train and its harness where your agent runs. Then describe the two runs you want to compare.

01 / Install

Set up once. Use your own agent.

Install two packages, each in the environment where it runs.

Install OpGuard + harness

# Training environment · Linux
python -m pip install "setuptools>=77.0.3" "packaging>=24.2" wheel ninja
python -m pip install --no-build-isolation --no-binary=opguard opguard

# Agent environment
python -m pip install opguard-harness

Python 3.10+. The training environment needs CUDA-enabled PyTorch and CUDA/CUPTI development tools.

Installation guide ↗

02 / Ask your agent

Two runs and a question.

Use natural language in your coding agent, such as Codex or Claude Code.

You → your coding agent

Use OpGuard to investigate why these two training runs differ: ./run-a/train.py and ./run-b/train.py.

My training Python is /path/to/train-env/bin/python.

Read `opguard-harness tools` for the available tools, and explain your findings with evidence.

Replace the paths with your own. Include the launcher arguments and configuration each run needs.

Agent integration guide ↗

03 / Example report

Follow the evidence to the cause.

A useful report connects the observed difference, a causal mechanism and a test.

Likely root cause

Gradient buffer reused before reduction completes.

A later write can reuse storage while the reduction stream is still reading an earlier bucket.

How to validate

Check stream completion before buffer reuse, or add a completion dependency and compare the runs again.

Follow the source

Observed norm append L1737 ↗all_norms.append
Shared storage L997 ↗self.ipg_buffer[self.ipg_index]
Reusing write L998 ↗new_grad_tensor.copy_

Expressions from the captured DeepSpeed source. Open a line to see its full context.

Explore the full diagnosis example ↓

Diagnosis demo

From a symptom to a source-level explanation.

In this recorded case, the loss still matches when the gradient norm starts to differ. Compare the log clues with the operation-level evidence and source used to investigate the cause.

Standard logs

Expand ↗

The symptom and a configuration clue.

Step 16 · rank 0
run1: loss 0.0467894077
      grad_norm 0.00197029114
run2: loss 0.0467894077
      grad_norm 0.00204745238

overlap_comm: False / True
ZeRO stage: 1 / 1
…

OpGuard output

Expand ↗

A specific difference to trace back through code.

First observed difference · rank 0
get_grad_norm_direct · None.append#2246
scalar input signatures:
  run1: [0, 1077945344, 0, 0]
  run2: [0, 1077804032, 0, 0]
Unaligned operations before boundary:
  run1: 0 · run2: 33
…

First-diff source: run1:1734 · run2:1736

Actual root cause

A gradient buffer is overwritten on the default stream before its reduction on another stream finishes. Details ↗

Explore how OpGuard exposes the evidence.

Open the trace explorer → See production results ↓

This recorded example uses precomputed, reviewed explanations from the logs, trace and captured source. It illustrates an investigation; it is not a live agent run or an accuracy benchmark.

Production Proven

Days of debugging, reduced to minutes

Deployed at ByteDance, OpGuard diagnosed kernel races, framework mismatches, and hardware-level corruptions that existing checks missed — including a five-day embedding backward race localized in under five minutes.

Production bug cases: manual root-cause time vs OpGuard (log scale). OpGuard stays at or under the 30-minute line.

OpGuard changes ByteDance’s internal debugging workflow in production training

Before

Manually compare two runs
Guess which subsystem to check

With OpGuard

Suspect run

Reference run

First mismatch: a reliable debugging pivot

Then inspect with the visualizer — tensor diffs, call stack, and source at the pivot.

Bitwise alignment enables broader comparisons

No “golden baseline” required

Compare equivalent computation across stacks, configs, and commits — pick whatever reference run is available for the bug.

Where are reference runs selected?

Self-replay 45%
Config tweak 25%
Cross-framework 20%
Stable commit 10%

Expert testimony

Developer A

We had been chasing the wrong subsystem for almost a week. OpGuard showed the exact kernel in under fifteen minutes.

Developer B

Without OpGuard, we would never have noticed that the drift originated in a single row race. The loss precision is too low and gives us no clue where to start.

Diverse and tricky root causes

Wrong model implementation / logic 7 cases (35%)

Race conditions 5 cases (25%)

Unexpected numerical drift 6 cases (30%)

Hardware faults 2 cases (10%)

20 Production Cases by Root-cause Categories

7 cases
(35%) 5 cases
(25%) 6 cases
(30%) 2 cases
(10%)

OpGuard supports major production-level training stacks

Internal: text and VLM pre-training, post-training framework

Open source: Megatron-LM, DeepSpeed, GPT-Neox, veRL, etc.

NVIDIA Megatron-LM DeepSpeed PyTorch 2.0 PyTorch Compiler veScale veRL PyTorch FSDP GPT-Neox Transformers

Explore the evidence

Go deeper in the trace explorer.

Explore the original demo_case5 capture, a separate recording of the same DeepSpeed case. Start at an observed difference, compare tensor fingerprints, and open its call stack and source. This is the evidence your agent can query during an investigation.

Initializing trace viewer…

Key Abstraction

Our key abstraction: bitwise alignment

Compare computations at tensor boundaries, then investigate where they first differ. We’re also expanding beyond bitwise toward more general, practical alignment.

Given the two training runs, view them as a sequence of “points”

Compare the tensors at each point. Must match bit by bit.

First mismatch point becomes a very clean pivot for debugging.

Buggy run Reference run

fp32

fp16

fp8

fp16

How It Works

OpGuard Workflow

Determinism control → preflight → guarded replay → offline alignment. Watch the build-up one step at a time.

1 Determinism Control

2 Preflight: (~ 5 iters) Discover Stable Op

3 Guarded Replay: Fingerprint at Op boundary

4 Offline Alignment

Other Compute

Attention

Suspect Run

Reference Run

Other Compute

Attention

Publication

Cite our paper

OpGuard was presented at OSDI ’26 and is a Best Paper Award nominee. If you use OpGuard in your research or systems work, please cite the following paper.

OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training

Ziming Zhou, Yinjie Zhao, Hang Zhu, Wenxiao Wang, Zhihao Bai, Yun Zhang, Shuguang Wang, Haibin Lin, Peng Huang

Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’26), Seattle, WA, USA, July 2026. USENIX Association.

Read the paper (PDF) OSDI ’26 page

BibTeX

@inproceedings{OpGuard2026OSDI,
  author = {Zhou, Ziming and Zhao, Yinjie and Zhu, Hang and Wang, Wenxiao and Bai, Zhihao and Zhang, Yun and Wang, Shuguang and Lin, Haibin and Huang, Peng},
  title = {{OpGuard}: Bitwise Alignment for Precise and General Debugging of Production {LLM} Training},
  booktitle = {Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation},
  series = {OSDI '26},
  month = {July},
  year = {2026},
  address = {Seattle, WA, USA},
  publisher = {USENIX Association},
}

Debug training differences with your AI agent.

Two runs. One prompt.

Set up once. Use your own agent.

Two runs and a question.

Follow the evidence to the cause.

Gradient buffer reused before reduction completes.

Follow the source

From a symptom to a source-level explanation.

Days of debugging, reduced to minutes

OpGuard changes ByteDance’s internal debugging workflow in production training

Bitwise alignment enables broader comparisons

Where are reference runs selected?

Expert testimony

Diverse and tricky root causes

20 Production Cases by Root-cause Categories

OpGuard supports major production-level training stacks

Go deeper in the trace explorer.

Our key abstraction: bitwise alignment

Given the two training runs, view them as a sequence of “points”

Compare the tensors at each point. Must match bit by bit.

First mismatch point becomes a very clean pivot for debugging.

OpGuard Workflow

1 Determinism Control

2 Preflight: (~ 5 iters) Discover Stable Op

3 Guarded Replay: Fingerprint at Op boundary

4 Offline Alignment

Cite our paper

Debug training differences
with your AI agent.