Files
blackbird/aibox/tests
qing 0246943165 feat(aibox): failover observability, timeout budget and Retry-After (L4c)
Observability: FailoverClient.get_stats() exposes aggregate counters
(requests / failovers / cooldown_skips / exhausted, snapshot copy);
new on_failover(from_name, to_name, reason) callback fires on every
failover decision — to_name=None marks chain exhaustion — for future
admin-panel/frontend wiring, wrapped so callback exceptions never
affect the request path.

Timeout budget: profile field "failover_timeout_budget" caps the total
seconds spent walking the chain (default unlimited; values <= 0 are
rejected to None with a warning). Once elapsed, remaining profiles are
skipped — prevents worst-case N x timeout stacking on longer chains.
Exhaustion is counted once per call via a single end-of-loop increment
(an initial double count at the budget branch was caught by the new
budget test and folded into the unified counter).

Retry-After: TransientLLMError now carries retry_after parsed from the
numeric Retry-After response header (non-numeric values like HTTP-dates
are ignored); FailoverClient._mark_fail uses it to extend that
profile's cooldown beyond the default 60s, respecting server-side
throttling on 429s. _fail_at entries became (fail_time, cooldown)
tuples; three existing tests that poke the internal map were updated.

Config: create_llm_client passes profile.failover_timeout_budget into
the FailoverClient constructor.

Tests: new TestL4cObservabilityAndBudget (13 cases: stats on
success/exhaustion/cooldown-skip, snapshot copy, hook called with
chain-tail None, hook exception safety, budget stops chain via mocked
monotonic, invalid budget ignored, config field plumbing, default
no-budget regression, Retry-After parse/ignore/extend-cooldown). Full
suite 355 passed.
2026-08-18 00:52:39 +08:00
..