Deterministic, judge-free analysis of tool-calling agent traces. Hard defects fail CI; events and candidates are shown for review, never asserted — no second model ever judges the trace.
One planted instance of each defect, clean controls, and legitimate-but-suspicious cases (a real retry, a value transform, a generated key). Each case asserts an expected outcome; the linter must match it.
| Result | Case | Scenario | Expected | Findings |
|---|---|---|---|---|
| PASS | r1_schema_violationplanted | A tool call whose arguments violate the declared JSON Schema (int, not string). | R1 hard_defect | R1+4 suppressed |
| PASS | r2a_tool_errorplanted | A tool returns a structured error (HTTP 500) — a hard event. | R2a hard_event | R2aR2bR3+3 suppressed |
| PASS | r2b_error_consumedplanted | A value from an errored result is reused as an argument to a side-effecting call. | R2b hard_defect | R2aR2b+2 suppressed |
| PASS | r3_hallucination_candidateplanted | An argument absent from provenance, with no schema annotation → candidate. | R3 candidate (not hard) | R2aR2bR3+2 suppressed |
| PASS | r3_hallucination_hardplanted | The same absent argument, but the schema declares the field 'provided' → hard_defect. | R3 hard_defect | R2aR2bR3+2 suppressed |
| PASS | r4_loopplanted | The agent repeats an identical failing call with no progress → loop candidate. | R4 loop candidate | R2aR2aR2aR2bR4R5 |
| PASS | r5_redundantplanted | An identical read repeated after unrelated work, with no mutation between. | R5 redundant candidate | R5+1 suppressed |
| PASS | clean_controlcontrol | A correct order-cancellation run — the linter must be silent. | no active findings | clean+1 suppressed |
| PASS | suspicious_retrysuspicious | A transient error followed by a successful retry — legitimate, not a loop. | no loop, no hard_defect | R2a+2 suppressed |
| PASS | suspicious_transformsuspicious | A legitimate reformat of a user-provided value (1,234.56 becomes 1234.56). | no R3 | clean+5 suppressed |
| PASS | suspicious_generated_keysuspicious | An idempotency key absent from context but declared 'generated' → not flagged. | no R3 | clean+4 suppressed |
Each agent is run against injected faults; recovery is scored against a deterministic success oracle, with a 95% Wilson CI.
cancel-if-not-shipped-robustcancel-if-not-shipped-buggy