Skip to content

Tutorial 4: Persistence and Human Checkpoints

Gates and recovery operate within a single engine.run() call. This tutorial covers two things that live outside it: state that survives a restart, and a place for a human to actually stop the loop.

The Amnesiac Loop

From the paper this project is built on:

The loop discovers work, does it, then forgets. The fix: a state file on disk - the agent forgets, the repo does not.

LoopStatePersistence writes LoopState to disk so a crashed or interrupted run can pick back up instead of starting over blind.

from loop_engine.persistence import LoopStatePersistence, PersistenceConfig

persistence = LoopStatePersistence(PersistenceConfig(state_dir="my_state", format="json"))

saved_path = persistence.save(engine.state, trace_id="demo-run-1")
print("saved to:", saved_path.name)

reloaded = persistence.load("demo-run-1")
print("reloaded iteration:", reloaded.current_iteration)
print("reloaded status:", reloaded.status.name)

Real, unedited output (continuing the loop from Tutorial 1):

saved to: loop_state_demo-run-1.json
reloaded iteration: 1
reloaded status: TERMINATED

reloaded status says TERMINATED, not COMPLETED - that's not a typo. LoopState.status and LoopResult.status are two different fields: LoopState.status records why the run stopped (a configured terminator firing sets TERMINATED), while LoopResult.status is _build_result()'s derived final verdict (COMPLETED, in this case) built from execution_state. Persist and inspect engine.state, not result, when you need to resume.

The JSON round-trip is intentionally partial

_from_json restores status, execution_state, current_iteration, and the component call counters - it does not reconstruct current_plan, failures, or recoveries from disk (the code says so directly: _to_json/_from_json are "a simplified version"). If your resume logic needs the full plan back, either keep the LoopState object alive in memory across the interruption, or extend _to_json/_from_json for your Step/Failure shapes before relying on this for a real crash-recovery path.

For simpler runs where you mainly want a human-readable audit trail rather than a byte-perfect resume, format="markdown" (the default) renders the plan, failures, and recoveries as tables instead - readable in any editor, not meant to be parsed back.

Keep one door open

From the same section of the paper:

The human did not leave, but changed desks - from writing to reviewing. The loop can execute, but it cannot decide.

HumanCheckpoint doesn't run your loop for you; it decides when to stop and ask. CheckpointConfig declares the triggers:

from loop_engine.checkpoint import HumanCheckpoint, CheckpointConfig, CheckpointTrigger

checkpoint = HumanCheckpoint(CheckpointConfig(
    trigger_on_major_failure=True,
    save_checkpoints=False,
))

Call should_pause(state) at whatever point in your loop you want to check - after engine.run(), or from inside a custom component with access to LoopState:

trigger = checkpoint.should_pause(state)
print("no failures yet, should_pause:", trigger)

failure = Failure(type=FailureType.EXECUTION_ERROR, message="repeated tool failure")
failure.mark_terminal()
state.failures.append(failure)

trigger = checkpoint.should_pause(state)
print("after terminal failure, should_pause:", trigger)

Real, unedited output:

no failures yet, should_pause: None
after terminal failure, should_pause: CheckpointTrigger.ON_MAJOR_FAILURE

Once a trigger fires, present_for_review() turns the current state into a ReviewRequest a human can actually read:

review = checkpoint.present_for_review(state, trigger, proposed_action="Escalate to human")
print(review.to_markdown())

# Human Checkpoint: checkpoint_0001

**Triggered by:** ON_MAJOR_FAILURE
**Time:** 2026-08-04T16:51:32.180204
**Proposed Action:** Escalate to human

## State Summary

- **Iteration:** 0
- **Status:** PENDING
- **Failures:** 1
...

That markdown is meant to be dropped straight into a Slack message, a GitHub issue comment, or a CLI prompt - ReviewResponse (APPROVE / REJECT / RETRY / MODIFY / PAUSE) is what comes back once a human actually decides. This is also exactly the mechanism RequestHumanHandler uses from Tutorial 2's HUMAN_HANDOFF recovery strategy - a checkpoint isn't a separate system bolted on top of recovery, it's one of recovery's own escalation paths.

Where this fits together

You now have all four pieces from this series:

  • Tutorial 1 - the six components and the state machine
  • Tutorial 2 - failures, strategies, and recovering without losing the loop
  • Tutorial 3 - catching problems before they reach a model
  • Tutorial 4 - surviving a restart, and knowing when to stop and ask

For a fully worked, larger example that ties recovery and the state machine together end to end, run examples/deterministic_multistep_loop.py. For wiring a loop into a scheduled, CLI-managed project instead of a standalone script, see the quickstart and the daily-triage pattern.