// eval gates
Quality gates that block bad deploys
Score runs against your own quality criteria with an LLM judge, then gate your CI/CD pipeline on the result, so regressions fail the build instead of reaching users.
LLM-as-judge
Define weighted criteria in plain language; a judge scores each run against them using the full conversation, not just metadata.
Auto eval-rules
Automatically evaluate incoming runs and trigger webhooks or emails the moment a score drops below your bar.
CI gate
retrace eval gate exits non-zero when a run scores below your threshold, drop it into any pipeline to block a bad deploy.
Suggested gates
Retrace studies clusters of past failures and proposes ready-made eval criteria that would have caught them.
Datasets
Turn real runs into datasets and run batch evaluations to track quality over time.
Offline regression gates
Diff a candidate run against a recorded golden run completely offline, with per-step tolerance and flaky-step detection.
- name: Eval Gate
run: retrace eval gate --evaluation $EVAL_ID --trace $TRACE_ID --threshold 0.8
env:
RETRACE_API_KEY: ${{ secrets.RETRACE_API_KEY }}Fail the build when a run scores below your threshold.
REST API
- POST /api/v1/evaluations: create an evaluation
- POST /api/v1/evaluations/:id/run: run it on traces
- POST /api/v1/evaluations/:id/gate: pass/fail gate
- POST/GET/DELETE /api/v1/eval-rules: automatic eval rules
More of the platform