Prove the Fix
Re-run a failed trace, diff it against the original, and get a verdict — did your fix actually work?
Prove the Fix
You changed a prompt, swapped a model, or patched a tool — but did it actually fix the failure? Prove the fix re-runs the trace, step-aligns a first-divergence diff against the original, and a judge returns a verdict.
Verdicts
| Verdict | Meaning |
|---|---|
improved | The fix did better (e.g. the original failed and the re-run completed, or the judge prefers the new output). |
regressed | The fix did worse. |
unchanged | No meaningful behavioral difference. |
inconclusive | The judge couldn't decide (or the replay failed). |
From the dashboard
Open a trace and click Verify fix (under the detections panel). You'll see the verdict plus a step-by-step diff with the divergent steps highlighted and the first divergence pinpointed.
From the API
curl -X POST https://api.retraceai.tech/api/v1/traces/$TRACE_ID/verify-fix \
-H "x-retrace-key: rt_live_..." -H "Content-Type: application/json" -d '{}'Returns { verification_id, verdict, summary, diff, resultTraceId, ... }. GET /api/v1/traces/:id/verify-fix lists past verifications.
Hypothesis testing (typed substitutions)
Instead of "re-run with current code", pass a substitution set to ask what-if questions — each is metered exactly like a verify-fix run:
-d '{ "substitutions": [ { "kind": "model", "value": "gpt-4o-mini" } ] }' # would a cheaper model have passed?
-d '{ "substitutions": [ { "kind": "tool_output", "span_id": "...", "value": "{...}" } ] }' # what if the tool returned X?Supported kinds: prompt, message, tool_output, model, temperature, router_choice. The substitution set is persisted with the verdict so any result is reproducible.
From the CLI / CI
retrace traces verify-fix <trace-id>Exit codes (stable — they're an API): 0 = improved or unchanged (pass), 1 = regressed (fail), 2 = inconclusive. Use a non-zero exit to fail a CI job on regression.
Plans
Prove-the-fix is metered on the fix_verification plan key and also consumes one fork replay. The Free plan has 0 verifications (Pro: 200/mo, Teams: 1,000/mo). The Fork & Replay add-on grants the replay half only — verify-fix itself requires a paid plan.