Failures and repair

Disks die, nodes reboot and networks split; WarrenFS treats each of these as routine. This page walks through a node failure minute by minute with the settings that govern each step, then covers bit rot, leftovers, and the failures two copies cannot absorb.

A node goes quiet

When	What happens	Setting
within 2 s	The other instances notice the missing health answers (cluster mode; kubernetes mode renews Leases every 10 s).	fixed
until `readonlyAfter`	Reads of the node's volumes come from the other copy. A write that picks one of its volumes fails with `503` and is retried, usually into another volume.
after `readonlyAfter`	The node's volumes take no writes. New writes go to volumes whose replicas are all healthy. Nothing in the map changes.	`repair.readonlyAfter`, default 1 min
after `repairAfter`	The coordinator puts another node in the silent one's place, as a `syncing` replica, in every volume and name shard it held.	`repair.repairAfter`, default 15 min
during repair	The new replica copies the index and the files from the surviving copy, checks each file against the source's checksum, replays the deletes made meanwhile, and reports the volume filled in its heartbeat. The coordinator makes it `active`; the volume takes writes again.	full speed

readonlyAfter must be at least 10 seconds in cluster mode (heartbeats every 2 s; 15 s is a good value) and at least 30 seconds in kubernetes mode. The lower it is, the sooner writes stop trying a dead node.

Repair runs at full speed, not at the rebalancing rate: until it finishes, the affected volumes have one copy. Because volumes are capped at 100 GB and spread over many node pairs, a lost node's volumes are re-copied in parallel, from and to many disks.

When the node comes back

Before repairAfter (a reboot, a kernel upgrade): its volumes take writes again as soon as its heartbeat is fresh. It pulls the deletes and name changes it missed from its peers, every minute, and serves as before. Nothing was copied.
After repairAfter: it has been replaced. It drops the copies the map no longer gives it, but only once each of those volumes has its full number of active copies elsewhere, so a stale map can never cost data.

What keeps working during a failure

Reads. Any instance reads from any replica: healthy ones first, silent ones as a last resort.
Writes. As long as some volume has all its replicas healthy. The coordinator keeps every healthy node in at least one writable volume and creates new ones on the healthy nodes.
Names. Every node holds every name shard. A name write goes to every healthy copy and skips the silent one, which catches up when it is back.
Deletes. One reachable replica is enough; the other picks the delete up later.
Mounts. The mount and the CSI driver retry for up to 30 seconds on 503 or an unreachable local instance, so a restart of the instance on the node is a pause, not an error.

Two nodes down

This is the limit of two copies, stated plainly. Any two nodes may hold both replicas of some volume. If exactly those two fail together, that volume is unreachable until one of them comes back, and lost if both disks are lost.

What still holds when two nodes are down:

names: with a copy on every node, lookups and name writes go on;
every volume that has a replica on a surviving node;
writes, into volumes whose replicas are both on surviving nodes.

What reduces the risk: the window in which a second failure hurts is repairAfter plus the time to copy the volumes, so keep repairAfter short and the private network fast. If two simultaneous failures are a risk you cannot take, WarrenFS with two copies is not enough on its own; keep backups elsewhere.

Losing the Raft majority

With three voters, losing two (or three of five) leaves the Raft group without a leader. Then:

the map cannot change: no new volumes, no sealing, no repairs, no moves, no drains;
reads and writes go on against the map every instance already holds, in volumes whose replicas are healthy.

Every instance then reports warrenfs_raft_has_leader 0, and warrenfs status says raft leader NONE: the map cannot change until a quorum of voters is back. Bring the voters back. Put voters on nodes that do not fail together, and use five when you have the nodes for it.

Bit rot: the scrub

Every scrub.every (a week) each instance re-reads every file it holds, at most scrub.maxRate (50 MB/s), and compares it with its SHA-256. A missing or corrupt copy is fetched from another active replica, checked, and swapped in inside an index transaction that first checks the entry is still live and unchanged, so a file deleted meanwhile stays deleted.

At 50 MB/s a node holding 1 TB needs about 5.6 hours per pass. A node's first pass starts an hour after it comes up. The counters and the time of the last full pass are in /metrics (warrenfs_scrub_*, see Metrics).

Leftovers: orphan GC

Once a day, files older than 24 hours that nothing can reach any more are removed:

on every node, files in a volume that its index does not know (a crash between writing a file and recording it);
by the coordinator, named files that no name points at (a write whose name was never recorded, an overwrite whose old file could not be deleted). It reads every name shard first, and deletes nothing if any shard cannot be read.

Renamed files and files written by key are left alone.

What WarrenFS does not protect against

Deleting the wrong file. A delete is final: there are no versions and no trash.
Both nodes of a pair failing for good. See Two nodes down.
An app writing bad data. Checksums prove that the copies are identical, not that they are right.
Needing a backup. Replication is not a backup. Keep backups of what matters outside the cluster.

Next: Names and the mount.