Day two

Running WarrenFS after day one is a short list: read warrenfs status, drain a node before it goes, add nodes when space runs low, alert on a handful of metrics, and upgrade with a rolling restart. Repair, scrub and rebalancing run by themselves.

Status

warrenfs status prints the cluster as the instance on this node sees it: space, nodes, volumes and, in cluster mode, the Raft group. In Kubernetes, run it in a warrenfs pod (kubectl exec -n warrenfs <pod> -- /warrenfs status); on bare metal, with the machine's environment (see Bare metal).

warrenfs cluster "warrenfs": 4 of 4 nodes healthy, 2 copies of everything

NODE   STATUS  DISK    FREE    WARRENFS  CACHE   FILES  VOLUMES  SHARDS  SEEN
node1  ok      1.9 TB  1.2 TB  441.2 GB  3.4 GB  70913  6        64      1s ago
node2  ok      1.9 TB  1.1 TB  452.6 GB  2.9 GB  71530  6        64      0s ago
node3  ok      1.5 TB  900 GB  425.9 GB  3.1 GB  70488  6        64      1s ago
node4  ok      1.5 TB  900 GB  428.1 GB  1.8 GB  70973  6        64      2s ago

used           873.4 GB  in 141952 files, counted once
used on disk   1.7 TB    all 2 copies and the indexes
free on disk   4.1 TB    on the healthy nodes
free for data  2 TB      new data that fits with 2 copies and 20 GB reserve per node

volumes: 12 for files (4 writable, 8 sealed), 64 name shards; 0 degraded, 0 replicas syncing

volume map: raft; raft leader node2 (as node1 sees it)
raft members: node1 voter, node2 voter, node3 voter, node4

--json prints the same as JSON, for scripts. Every number is also in /metrics.

Drain a node

Before a node leaves for good, or its disk is replaced, move its data off:

warrenfs drain node3

node3: drain done

The command returns once the drain is recorded; the moves then run in the background. The node's volumes move to other nodes one at a time, it leaves the name shards, and it gets no new volumes. Watch its VOLUMES column reach 0. warrenfs undrain node3 stops it; volumes already moved stay where they are.

In kubernetes mode a drain is the node label warrenfs/drain=true instead.

Remove a node that is gone

warrenfs remove node3

takes a node that is gone for good out of the Raft group. Its volumes are repaired elsewhere like those of any node silent for longer than repairAfter. Drain first when the node is still there; remove when it is not.

Add a node

In Kubernetes the DaemonSet starts an instance on the new node; on bare metal, give it the config, its identity and the CA and start the service (Bare metal). Then, without any command:

1. The Raft leader adds it as a non-voting member once it answers health checks.
2. It gets a copy of every name shard, filled like a repair.
3. The coordinator puts it in a writable volume, on the nodes with the most free space, so new data starts landing on it at once.
4. If the usage gap between the fullest and the emptiest node stays above 10 points for 30 minutes, the rebalancer starts moving volumes to it.

Rebalancing

Always on, with these rules:

• Repair beats rebalance. A move starts only when nothing is being repaired, every node is healthy, and no other move runs. One move at a time.
• Drained nodes first. Otherwise a volume moves from the fullest node to the emptiest when the gap stays above rebalance.spreadThreshold (10 points) for rebalance.holdTime (30 minutes), and only if the move takes at least one point off the fullest node, so small volumes are not shuffled around.
• Never below two copies. The target is filled as an extra syncing replica and checked; only then does the source become draining, and it keeps serving reads for two minutes while every instance picks up the new map.
• Paced. A move copies at most rebalance.maxRate (50 MB/s), so live traffic keeps the disks and the network. A volume being moved takes no writes; reads go on.

A 100 GB volume moves in about 35 minutes at that rate. Set rebalance.enabled: false to stop automatic moves; drains still need it on.

Scrub

Each instance re-reads every file it holds once a week, at most 50 MB/s, and repairs bad copies from a peer. It needs no attention unless warrenfs_scrub_failed_total rises: that is a bad copy no peer could replace. See Failures and repair.

Alerts worth having

Every instance computes the cluster-wide metrics from its own copy of the map, so they come once per instance; take the max across instances.

Pick the space threshold to match how fast you write and how long it takes you to add a disk.

Switch on TLS between instances

Two rolling restarts, no downtime:

1. Make a CA (warrenfs peer-ca DIR), put ca.crt and ca.key in every instance's secrets as WARRENFS_PEER_CA_CERT and WARRENFS_PEER_CA_KEY (the PEM, or a file path), and set peerTLS: on. Restarted instances accept TLS and plain, and dial TLS; they fall back to plain only for the peers not restarted yet.
2. Set peerTLS: required. Every instance already accepts TLS, so nothing breaks while they restart; afterwards a peer request without TLS gets 403.

Raft traffic is included. The apps' traffic (the CSI mount, the HTTP API) stays plain HTTP to the instance on the same node, on the same port, and never crosses the network.

Upgrade

A rolling restart, one node at a time. While an instance restarts:

• its volumes keep serving reads from their other copies;
• writes that pick one of its volumes get 503 and are retried; if the restart takes longer than readonlyAfter, its volumes stop taking writes until it is back;
• mounts on that node wait and retry for up to 30 seconds, and the CSI plugin, a separate DaemonSet, keeps every mount in place;
• in cluster mode, a stopping Raft leader hands leadership over first, so the coordinator moves without an election timeout.

Upgrade the CSI plugin separately and on purpose: when it restarts, it mounts every volume again.

Move from kubernetes mode to cluster mode

No downtime, and nothing copied but the map:

1. Deploy a version with cluster mode, config unchanged. It runs exactly as before.
2. Set mode: cluster, migrateFromKubernetes: true, peers and voters, and restart the instances one at a time. Until every instance with a recent Lease runs in cluster mode, the map stays in Kubernetes and the coordinator is still elected through the Lease, so old and new instances work together. The new ones meanwhile form the Raft group and check each other's health.
3. Once every instance is a Raft member and answers health checks, the coordinator reads the map from the API server and commits it to Raft in one command, drain labels included. Every instance switches when it applies that command, and the coordinator logs volume map: moved to raft. warrenfs status then shows volume map: raft.
4. Later, migrateFromKubernetes can come out of the config. The Volume resources and Leases stay as they were, frozen, until you delete them; the RBAC for them can go too.

Going back to a version without cluster mode after step 3 is safe only while the map has not changed since the move: the old version reads the frozen custom resources.