OpGuard

OSDI ’26 Best Paper Award Nominee

Catch LLM training bugs
at the first divergent op

OpGuard uses bitwise alignment to compare training runs at tensor boundaries, turning vague loss-curve anomalies into precise, actionable evidence. We’re expanding beyond bitwise toward more general, practical alignment.

Explore Live Trace Source Code How It Works

Open source · Apache-2.0

5 min vs. 5 days to localize
20+ production issues diagnosed
bit-exact tensor fingerprint alignment

Find the root cause.

Standard logs

Expand ↗
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
…
Cached diagnosis · no model callLogs + recorded versions · 3 reviewed examples

OpGuard output

Expand ↗
First observed difference · rank 0
get_grad_norm_direct · None.append#2246
scalar input signatures:
  run1: [0, 1077945344, 0, 0]
  run2: [0, 1077804032, 0, 0]
Unaligned operations before boundary:
  run1: 0 · run2: 33
…

First-diff source: run1:1734 · run2:1736

Cached diagnosis · no model callTrace + logs + captured source · 3 reviewed examples
Actual root cause

A gradient buffer is overwritten on the default stream before its reduction on another stream finishes. Details ↗

Reviewed example diagnoses for this fixed case. Each click randomly selects a cached variation, with no live model call or inference charge. The OpGuard explanations use the trace, recorded logs and linked full source.

These are curated explanations, not unedited model responses or an accuracy benchmark. Expand opens the recorded evidence; source links open the captured code at the cited line.

Key Abstraction

Our key abstraction: bitwise alignment

Given the two training runs, view them as a sequence of “points”

Compare the tensors at each point. Must match bit by bit.

First mismatch point becomes a very clean pivot for debugging.

Buggy run Reference run
fp32
fp16
fp8
fp16
Interactive Demo

Global Alignment Trace

Inspect a real production debugging session. The first divergent operation marks where the bug begins — click any op to inspect tensor-level diffs and jump to source.

Loading trace…
demo_case5 Open-source issue

Initializing trace viewer…

How It Works

OpGuard Workflow

Determinism control → preflight → guarded replay → offline alignment. Watch the build-up one step at a time.

1 Determinism Control

2 Preflight: (~ 5 iters) Discover Stable Op

3 Guarded Replay: Fingerprint at Op boundary

4 Offline Alignment

V
Other Compute
K
Attention
Suspect Run
Reference Run
Other Compute
K
V
Attention
OpGuard Aligns Diverse Runtime Executions
Production Proven

Days of debugging, reduced to minutes

Deployed at ByteDance, OpGuard diagnosed kernel races, framework mismatches, and hardware-level corruptions that existing checks missed — including a five-day embedding backward race localized in under five minutes.

Debugging time comparison on a log scale 10 min 1 h 1 day 1 week 2 weeks 30 min Debugging time Emb-bwd (3080) CUDA stream FlashAttn det NCCL topo Scatter SDC MoE hist Emb-bwd fuse Offload sync Triton bwd QKV layout Optim resume

Production bug cases: manual root-cause time vs OpGuard (log scale). OpGuard stays at or under the 30-minute line.

OpGuard changes ByteDance’s internal debugging workflow in production training

Before
  • Manually compare two runs
  • Guess which subsystem to check
With OpGuard

Then inspect with the visualizer — tensor diffs, call stack, and source at the pivot.

Bitwise alignment enables broader comparisons

No “golden baseline” required

Compare equivalent computation across stacks, configs, and commits — pick whatever reference run is available for the bug.

Where are reference runs selected?

  • Self-replay 45%
  • Config tweak 25%
  • Cross-framework 20%
  • Stable commit 10%

Expert testimony

Developer A

We had been chasing the wrong subsystem for almost a week. OpGuard showed the exact kernel in under fifteen minutes.

Developer B

Without OpGuard, we would never have noticed that the drift originated in a single row race. The loss precision is too low and gives us no clue where to start.

Diverse and tricky root causes

Wrong model implementation / logic 7 cases (35%)
Race conditions 5 cases (25%)
Unexpected numerical drift 6 cases (30%)
Hardware faults 2 cases (10%)

20 Production Cases by Root-cause Categories

7 cases
(35%) 5 cases
(25%) 6 cases
(30%) 2 cases
(10%)

OpGuard supports major production-level training stacks

Internal: text and VLM pre-training, post-training framework

Open source: Megatron-LM, DeepSpeed, GPT-Neox, veRL, etc.

NVIDIA Megatron-LM DeepSpeed PyTorch 2.0 PyTorch Compiler veScale veRL PyTorch FSDP GPT-Neox Transformers
Publication

Cite our paper

OpGuard was presented at OSDI ’26 and is a Best Paper Award nominee. If you use OpGuard in your research or systems work, please cite the following paper.

OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training

Ziming Zhou, Yinjie Zhao, Hang Zhu, Wenxiao Wang, Zhihao Bai, Yun Zhang, Shuguang Wang, Haibin Lin, Peng Huang

Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’26), Seattle, WA, USA, July 2026. USENIX Association.

Read the paper (PDF) OSDI ’26 page
BibTeX 
@inproceedings{OpGuard2026OSDI,
  author = {Zhou, Ziming and Zhao, Yinjie and Zhu, Hang and Wang, Wenxiao and Bai, Zhihao and Zhang, Yun and Wang, Shuguang and Lin, Haibin and Huang, Peng},
  title = {{OpGuard}: Bitwise Alignment for Precise and General Debugging of Production {LLM} Training},
  booktitle = {Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation},
  series = {OSDI '26},
  month = {July},
  year = {2026},
  address = {Seattle, WA, USA},
  publisher = {USENIX Association},
}