Skip to content
← All work

2026

Agent reliability

An offline test harness for tool-using agents with execution traces, deterministic graders, and regression checks in CI.

Role
Sole engineer
Stack
Python, pytest, GitHub Actions, JSONL traces
Code
GitHub repository
Links

The problem

A successful agent run says little about how the system behaves when a tool fails or execution is interrupted. I wanted repeatable tests for failures such as timeouts, partial state, blocked writes, and regressions in tasks that previously passed.

The harness runs these cases offline with a scripted planner. Tool behavior, recovery, policy checks, and grading can therefore be tested deterministically before adding a live model.

How it’s built

  1. Tool router

    Validates schemas and handles timeouts and retries for sandboxed fs_read, fs_write, http_get and flaky_echo tools.

  2. Runtime

    Runs the planner loop, stores checkpoints and applies policy checks before side effects.

  3. Trace sink

    Records tool calls, retries, policy blocks, resumes and failure classes as JSONL events.

  4. Graders

    Assign deterministic pass or fail results and failure classes for tasks T1 through T6.

  5. Eval + CI

    The offline runner writes the evaluation report and CI compares it with a pinned baseline.

Tool calls pass through the runtime and are written to JSONL traces. Deterministic graders score each task, and CI compares the resulting report with a pinned baseline.

What it checks

  • Tool inputs are validated against schemas, with configured timeout and retry behavior.
  • Calls, retries, policy blocks and checkpoint resumes are recorded in the trace.
  • Each task receives a deterministic result and failure class.
  • CI compares the current report with a pinned baseline and fails when a regression is detected.

Task suite

Offline baseline generated 2026-09-13. All six harness tests pass. The suite evaluates runtime behavior, not model capability.

T1 File repairRead broken JSON, repair it, write idempotently
Pass
T2 API reconcileTwo HTTP calls reconciled into one report
Pass
T3 Flaky toolRecovery through router retries on injected failures
Pass
T4 Partial stateCheckpoint, injected crash, resume from the store
Pass
T5 Policy refusalUnsafe write blocked before it happens
Pass
T6 Cost budgetFinishes under token and cost counters
Pass

Decisions

Start with deterministic evaluation

Version 0 uses a scripted planner so the complete evaluation suite can run offline without API keys or model variance.

TradeoffThe current results evaluate the runtime and harness only. A live planner can later use the same router, trace and grader interfaces.

Track failure classes in CI

CI checks both the pass rate and the recorded failure class for each task against the pinned baseline.

TradeoffLatency is excluded from the baseline so results remain stable across machines. The current gate therefore does not detect latency regressions.

Run policy checks before tool execution

The runtime checks tool actions against policy before execution. Blocked actions are recorded in the trace and can be graded as expected refusals.

TradeoffThe current policy uses explicit allowlists. Adding a new tool requires adding its permitted behavior to the policy.

Limits

  • Version 0 uses a scripted planner, so the results do not measure LLM task performance.
  • The partial-state test raises an exception after checkpointing rather than terminating the operating-system process.
  • Cost counters use simulated token usage rather than provider billing data.