Tutorial 2: Recovery and Resilience

Tutorial 1 built a loop that could not fail. Real actors fail constantly: a tool call times out, a generated diff doesn't compile, an evaluator rejects a first draft. This tutorial makes a step fail on purpose and watches the engine recover.

Failures have a lifecycle, not just a message

When an Actor.execute() raises, LoopEngine doesn't just log it and stop. It creates a Failure record with its own state machine:

UNHANDLED -> RECOVERY_PLANNED -> RECOVERY_IN_PROGRESS -> RECOVERED
                                                      \-> RECOVERY_FAILED -> TERMINAL

Failure.can_recover() checks recoverable, current status, and recovery_attempts against max_recovery_attempts before the engine will try again - failures don't retry forever by accident.

Part A: the default - RETRY

Without a RECOVERY component registered, the engine always retries a recoverable failure. Here's an actor that fails once, then succeeds:

from loop_engine.components import Actor

class FlakyActor(Actor):
    """Fails on the first attempt, then succeeds."""

    def __init__(self):
        self.attempts = 0

    async def execute(self, step, observations, context):
        self.attempts += 1
        if self.attempts == 1:
            raise ValueError("simulated transient failure")
        return {"attempt": self.attempts, "result": "ok"}

    async def execute_direct(self, goal, context):
        return {"result": "ok"}

Register it exactly like Tutorial 1's WordCountActor (same SimplePlanner / SimpleObserver / HasResultEvaluator / SimpleTerminator cast), run it, and inspect result.failures / result.recoveries:

result = await engine.run(context)

print("status:", result.status.name)
print("iterations:", result.iterations)
print("failures recorded:", len(result.failures))
print("recoveries executed:", len(result.recoveries))
print("recovery strategy used:", [r.strategy.value for r in result.recoveries])
print("output:", result.output)

Real, unedited output:

status: COMPLETED
iterations: 2
failures recorded: 1
recoveries executed: 1
recovery strategy used: ['retry']
output: {'attempt': 2, 'result': 'ok'}

The first iteration recorded a failure and retried the step; the second iteration's attempt succeeded and the loop completed - two iterations, one recorded failure, one recovery, final output from the successful attempt.

Part B: six strategies, one registry

RETRY is one of six strategies RecoveryRegistry ships a real handler for - every one performs an actual state change, not just a label:

Strategy	Handler	What it does
RETRY	RetryHandler	Reset the step to READY, try again
BACKOFF	BackoffHandler	Like retry, with an escalating delay recorded as evidence
PLAN_REVISION	ReplanStepHandler	Replace the failed step with a corrected version, bump plan version
GRACEFUL_DEGRADATION	GracefulDegradationHandler	Mark the step SKIPPED so the plan can still complete
HUMAN_HANDOFF	RequestHumanHandler	Transition to WAITING_FOR_HUMAN and pause the loop
CIRCUIT_BREAK	TerminateHandler	Mark the failure terminal and end the loop as FAILED

Every handler also implements validate_postconditions() - the registry checks the handler actually did what it claimed (step really is READY, plan version really incremented, state.metadata really has the escalation record) before trusting a "success" result. See recovery/handlers.py for the full implementation.

Part C: choosing a strategy by failure type

Always retrying isn't always right - a security-relevant failure shouldn't just retry quietly. Register a ComponentType.RECOVERY component and the engine asks it to choose a strategy per failure instead of defaulting to RETRY. AdaptiveRecovery (shipped in loop_engine.components) is a type-aware example:

from loop_engine.components import AdaptiveRecovery
from loop_engine.types import Failure, FailureType, LoopContext

recovery = AdaptiveRecovery()
context = LoopContext(goal="demo")

for failure_type in [
    FailureType.TIMEOUT,
    FailureType.VERIFICATION_FAILED,
    FailureType.INVALID_OUTPUT,
    FailureType.GOAL_HIJACKING,
    FailureType.EXECUTION_ERROR,
]:
    failure = Failure(type=failure_type, message="demo failure")
    action = await recovery.recover(failure, state=None, context=context)
    print(f"  {failure_type.value:22s} -> {action.strategy.value}")

Real, unedited output:

  timeout                -> backoff
  verification_failed    -> plan_revision
  invalid_output         -> retry
  goal_hijacking         -> circuit_break
  execution_error        -> graceful_degradation

Note the asymmetry: a transient timeout backs off and retries, a rejected evaluation triggers a plan revision instead of blindly retrying the same approach, and GOAL_HIJACKING - a security-relevant failure type - goes straight to CIRCUIT_BREAK rather than getting more attempts. Register your own RECOVERY component to encode whatever escalation policy your domain needs; AdaptiveRecovery is a starting point, not the only option.

See the full state machine trace

examples/deterministic_multistep_loop.py runs a three-step plan where the middle step fails twice, gets recovered twice, and the loop still reaches COMPLETED - with an explicit, inspectable trace of every state transition:

python examples/deterministic_multistep_loop.py

What's next

Recovery decides what to do after something goes wrong. Tutorial 3 is about catching whole classes of problems before they ever reach an LLM call, using deterministic gates.