You are watching a recover via a Leader and Followers pattern, using a High-Water Mark to truncate a Write-Ahead Log , protected by a Lease and a Generation Clock .
That is the legacy of Unmesh Joshi. He taught us to see the clockwork. Unmesh Joshi is a Principal Consultant at ThoughtWorks and the author of the upcoming O'Reilly book, "Patterns of Distributed Systems." His pattern catalog is available at martinfowler.com. unmesh joshi patterns of distributed systems
These aren't abstract algorithms. They are concrete patterns with names, problem statements, solutions, and consequences. Let’s look under the hood. When you read Joshi’s work (collected on Martin Fowler’s website and in his upcoming O’Reilly book), you don't start with Byzantine Generals. You start with the gritty reality of what happens when a server dies. You are watching a recover via a Leader
Unmesh Joshi has effectively written the "Gang of Four" book for distributed systems. Unmesh Joshi is a Principal Consultant at ThoughtWorks
In the modern era of software engineering, we speak in superlatives. We boast about systems that span continents, handle millions of requests per second, and achieve "five-nines" of availability. Yet, for most engineers, the internals of these systems remain a black box—a magical realm of consensus algorithms, replication logs, and failure detectors.
In his famous essay, "The Pattern Language of Distributed Systems," he writes: "You don't choose a distributed system. You inherit its complexity. The patterns help you live with that complexity, not fight it." He treats distributed systems as a biological ecosystem. Patterns compete. "Heartbeat" is cheap but prone to false positives. "Lease" is safer but requires synchronized clocks (which you don't have). "Epoch" (or "Generation Number") is the safest, but it requires persistent storage.
In his writing, a "Heartbeat" isn't just a ping. It is a pattern with specific failure modes. What happens if the heartbeat is delayed by a garbage collection pause? The system might falsely declare a leader dead (a "false positive"). To fix this, you need the "Lease" pattern—a time-bound guarantee that prevents two leaders from existing simultaneously (the dreaded "split brain").
