Case study 02 / Routing resilience

OSPF Resiliency
Lab

A reproducible four-router lab that proves equal-cost routing, injects a silent core failure, and records how the network converges and recovers.

View source Verified run

Role: Lab design and automation
Stack: FRRouting · Containerlab · Python
Topology: Area 0 · 4 routers · 5 links
Status: End-to-end verified

The problem

Redundant paths on a topology diagram do not prove that routing will survive a failure.

A useful resiliency test needs a known baseline, a controlled fault, route and adjacency evidence, a reachability check, and a verified return to normal.

This lab makes that sequence repeatable. The same command builds the topology, checks both forwarding paths, pauses one core router, measures convergence, and confirms recovery.

Topology and policy

Two edge routers communicate through independent core routers, with equal OSPF cost through either side.

All links participate in OSPF area 0. Router IDs are explicit, transit interfaces use the point-to-point network type, and interfaces are passive unless an adjacency is required.

A higher-cost link joins the core routers. It remains available to OSPF but stays outside the normal equal-cost forwarding path between the edges.

Failure experiment

BaselineProve both paths
Wait for every expected Full adjacency, confirm two selected next hops, and test ICMP.
FaultPause core-a
Stop the router process while leaving its links present, creating a silent control-plane failure.
ConvergeMeasure failover
Time how long it takes to remove the failed path and verify traffic through core-b.
RecoverRestore ECMP
Resume core-a, then wait for all adjacencies and both next hops to return.

Engineering decisions

Silent failure injection

Pausing the router forces neighbors to detect control-plane silence through the dead timer instead of receiving an immediate link-down event.

Assertions over screenshots

The verifier reads FRRouting JSON, counts Full neighbors, checks selected next hops, and fails the run if an expectation is not met.

Fast, explicit timers

One-second Hello and three-second dead timers keep the experiment short while making the test assumptions visible in every configuration.

Reproducible dependencies

The FRRouting image and Containerlab release are pinned, and the complete topology, configs, tests, and runbook live with the code.

Measured result

2baseline next hops

2.800sfailed path removed

Passreachability during failure

6.094sboth paths restored

These values came from the first successful GitHub-hosted integration run. The committed JSON report records the next hops, neighbor counts, injected command, timings, reachability result, and recovery state.

Validation

11static tests

3Python versions in CI

4live FRR routers

1machine-readable report

GitHub Actions validates the address plan, topology, timers, routing policy, parser behavior, and dependency pins before deploying the live lab. The integration job then runs the complete experiment and always destroys the topology, even if a check fails.

Limits and next steps

A controlled lab result is evidence of behavior, not a production convergence guarantee.

The timings depend on the chosen timers and GitHub-hosted runner. The experiment observes one source and destination, and a paused container models control-plane silence rather than every physical or software failure mode.

Useful next steps are repeated-run distributions, packet-level capture during convergence, additional failure points, and an observability layer that tracks latency and reachability over time.

Next project

Observe the network,
end to end.

Read diagnostics case study

OSPF ResiliencyLab