Stage 8 · L38

Model circuit recovery and release cancelled probes

Core · original Session 8

Native checkpoint 08

Use the starter for this stage's focused examples. The cumulative transfer and solution belong at the stage-end capstone. Baseline checks pass; transfer checks initially fail. Downloads contain the matching native starter and solution for this stage.

A circuit breaker limits repeated work against a failing integration. Closed allows calls and tracks consecutive logical failures. After the threshold it opens and rejects calls until a cooldown. Then one half-open probe may test recovery. A successful probe closes; a failed or abandoned probe reopens.

Calls carry Ticket(generation, probe). Opening or completing a probe advances the generation, so a late success from an older call cannot close a newer open state. The asyncio.Lock protects these local state transitions. It does not coordinate independent workers or hosts; the breaker belongs to this application event loop.

The independent task is abandonment. Cancellation while a half-open probe awaits must reopen that generation and propagate CancelledError. Leaving half_open would strand the circuit with no possible future probe. The test cancels an actual pending Gateway lookup after a transport event, asserts cancellation reaches the caller, and later confirms a new probe becomes eligible.

Local admission/pool pressure uses abandonment and does not count as provider outage. A normal closed ticket's abandonment leaves failure history unchanged. A half-open ticket's abandonment reopens for cooldown, because the one probe did not establish recovery. Provider404 is valid reachability and calls success rather than failure.

Read the stale-result test and the exactly-one-probe assertion together. A state diagram alone cannot prove cancellation or concurrency behavior. This checkpoint's real async failure tests replace the earlier standalone breaker teaching sketch; it is still a fixture integration, not an installed live commercial-provider breaker.

Closed: calls allowed; Threshold: open/cooldown; Cooldown: one probe; alternative outcomes: Success / Close circuit / New generation or Fail/cancel / Reopen circuit / Release probe; A stale generation cannot change newer circuit state.

Follow the running code

Focused lesson example; see the end-of-stage capstone for the cumulative app · stage 08

closed → failures reach threshold → open
open → cooldown passes → one half-open probe
probe succeeds → closed; probe fails/abandons → open
Stale generation completion → no change to newer state.

Predict and observe this focused example using the concepts explained above. Its boundary is stated in the focused answer.

Guided lab

Read the explanation and predict the focused example’s outcome.
Cancel the owning half-open probe, then predict the state and next allowed probe. Explain why a stale success cannot reset newer failures.
Compare the observed outcome with the focused answer and state its boundary.

Expected: Abandonment under the lock reopens only the current owning probe generation and establishes a new cooldown; cancellation propagates. A later eligible probe can proceed. A stale generation result cannot close or reset a newer circuit. Local admission failure is separate from provider-outage failure. The full native resilience transfer appears only in this stage capstone.

Swallowing cancellation or stranding half_open breaks recovery; generations must guard old completions.

Focused exercise and answer

Complete this focused exercise before reading its answer. The full native transfer is introduced only at the end of the stage.

Your transfer task: Cancel the owning half-open probe, then predict the state and next allowed probe. Explain why a stale success cannot reset newer failures.

Cancel the owning half-open probe, then predict the state and next allowed probe. Explain why a stale success cannot reset newer failures.

Inspect the matching answer

This answer addresses the focused exercise above; the cumulative implementation is shown only after the stage prerequisites.

Abandonment under the lock reopens only the current owning probe generation and establishes a new cooldown; cancellation propagates. A later eligible probe can proceed. A stale generation result cannot close or reset a newer circuit. Local admission failure is separate from provider-outage failure. The full native resilience transfer appears only in this stage capstone.

Stage 08 capstone — after these prerequisites

Implement the breaker’s current-generation abandoned probe cleanup; cancellation propagates and local capacity is not provider outage.

Use the downloaded starter after completing this stage's focused exercises. The cumulative implementation below is a stage transfer answer, not an answer to an earlier lesson.

python run_checks.py --stage 08 --role starter --prepare
python run_checks.py --stage 08 --role starter
python run_checks.py --stage 08 --role starter --transfer

From this extracted starter: preparation and baseline pass; transfer initially fails only at the named unfinished target. After implementing it, rerun the same starter --transfer command and expect success.

Optional comparison in a separate solution directory

Optional comparison: download and extract this stage’s solution ZIP into a separate directory. Change your terminal into that extracted solution root (beside checkpoint.json and run_checks.py) before running the following commands. Your starter remains a starter even after you implement its task.

python run_checks.py --stage 08 --role solution --transfer

Inspect the cumulative capstone implementation

class CircuitBreaker:
    def __init__(self, threshold=2, cooldown=5.0, clock=time.monotonic):
        self.threshold, self.cooldown, self.clock = threshold, cooldown, clock
        self.state, self.failures, self.generation, self.until = "closed", 0, 0, 0.0
        self.lock = asyncio.Lock()

    async def allow(self):
        async with self.lock:
            if self.state == "closed":
                return Ticket(self.generation, False)
            if self.state == "open" and self.clock() >= self.until:
                self.state = "half_open"
                return Ticket(self.generation, True)
            raise DomainError(503, "provider_circuit_open", "Provider check is temporarily unavailable")

    async def success(self, ticket):
        async with self.lock:
            if ticket.generation != self.generation:
                return
            self.failures = 0
            if ticket.probe:
                self.state = "closed"
                self.generation += 1

    async def failure(self, ticket):
        async with self.lock:
            if ticket.generation != self.generation:
                return
            self.failures += 1
            if ticket.probe or self.failures >= self.threshold:
                self.state = "open"
                self.until = self.clock() + self.cooldown
                self.generation += 1

    async def abandon(self, ticket):
        async with self.lock:
            if ticket.probe and ticket.generation == self.generation:
                self.state = "open"
                self.until = self.clock() + self.cooldown
                self.generation += 1

Check your reasoning

What must happen when the only half-open probe is cancelled?

Show the explanation

Abandon its generation, reopen for cooldown and propagate cancellation so the caller retains control.

Reading progress

54 lessons remain open to guests. Marking a lesson read records reading only; it does not award assessment credit or a certificate.

Device reading marks require browser storage. Reading is always available.