Each Orbita partition has one owner and two full replicas. The owner serializes writes. Replicas exist both to preserve the WAL and to serve reads, because a read-optimized store cannot funnel every read through one node per range and still call itself scalable.
The obvious design is a partition-wide watermark: if a replica has applied the latest committed Lamport, it serves locally; otherwise it forwards to the owner. That design fails twice.
First, one write anywhere in the partition makes the replica stale for every key in the partition. Under sustained writes a replica is nearly always one entry behind, so almost every read forwards. The fan-out disappears exactly when load makes it useful.
Second, the replica’s knowledge of the committed watermark may itself be stale. Believing you are current is not the same as being current.
The replica should not ask whether it is current. The owner should tell it what became unsafe.
Invalidate the key on the traffic that already exists
Every write is already replicated as a WAL entry. When a replica receives an entry for key K, it marks Kunreadable before applying or acknowledging that entry. The message that makes a write durable is also the message that prevents a stale read. There is no additional healthy-path round trip.
Once the replica applies the entry, it clears the key from its invalid set. Reads for unrelated keys keep flowing throughout. The invalid set is bounded by replication lag, so its growth is also a useful signal that a replica is unhealthy.
Leases bound the slow-replica problem
Without leases, one unreachable replica would stall every write forever: the owner could not know whether it might still serve the old value. A replica therefore serves only while it holds a short read lease from the owner. When it becomes slow, the owner stops renewing the lease and waits for the remainder to expire. The delay is bounded, and the sick replica then leaves the read set.
The clocks need not be synchronized. The owner measures the lease from when it sent the heartbeat. The replica measures from when it received it, minus a safety margin, so the replica always gives up first.
The votes answer different questions
A client acknowledgement waits on two conditions. The entry must be durable on two of the three WALs. Separately, every replica whose read lease is still live must have acknowledged the key invalidation—or its lease must have expired.
In the healthy case, the same acknowledgement messages satisfy both conditions. Under failure, keeping the conditions separate matters. A durability quorum answers “will this write survive?” A coherence quorum answers “can anyone still hand out the old value?” Treating them as one question either weakens durability or makes the system wait forever.
This is the sort of design that looks straightforward on a diagram and fails silently in an implementation. Orbita’s simulator targets the invalid set, lease expiry, message gaps, and ownership changes directly. Conservative uncertainty costs one forwarding hop; optimistic uncertainty breaks linearizability.