Architecture

WarrenFS is one Go binary that runs once on every node. Each instance stores files on its node's own disk, writes every file to a second node, and keeps a copy of one shared volume map; one instance at a time coordinates. There is no metadata server to lose, no gateway in the data path, and no raw block device.

One binary on every node

warrenfs serve runs on each node: as a DaemonSet with host networking in Kubernetes, or as a systemd service on bare metal. Every instance is a storage node:

• it owns the node's data directory (/var/lib/warrenfs by default);
• it accepts writes from apps on the same node and streams them to the volume's replicas;
• it serves the files it holds, and fetches the others from a peer;
• it scrubs its files, collects orphans and pulls the deletes it missed.

One instance at a time is also the coordinator: the Raft leader in cluster mode, or the holder of a Lease in kubernetes mode. The coordinator seals full volumes, creates new ones, replaces silent replicas and moves volumes. It is a role, not a machine: when its instance goes away, another one takes over.

The same binary is also the client side: warrenfs mount (a FUSE mount of a bucket), warrenfs csi (the Kubernetes CSI node plugin) and the admin commands (status, drain, remove, peer-ca). See the CLI reference.

A public read path is designed but not built yet: an entry point that serves small files and redirects large ones, with a signed URL, to a node that holds them, so video leaves through that node's own uplink. Its settings (public.*) exist; nothing listens on them. Today every read goes through the internal API, the mount or the CSI driver.

Three modes

Cluster mode is the one to run: WarrenFS then depends on nothing outside itself, not even the Kubernetes API. The built-in default is still kubernetes, so set mode explicitly. Moving an existing cluster over takes no downtime and copies nothing but the map; see Day two.

The internal listener

Each instance listens on one port, 7480 by default, bound to the node's private IP. Everything goes through it:

• apps on the node: the HTTP API (/v1/files, /v1/names), the mount and the CSI driver, with a bearer token per app;
• other instances: replication, repair, health checks and Raft, under /peer/, with the peer token;
• monitoring: /metrics and /healthz, without a token.

With peerTLS on, a connection that starts with a TLS handshake is mutual TLS, checked against the cluster CA; any other connection is plain HTTP for the pods on the node. Same port, no second listener, and the apps' traffic never leaves the node. Keep port 7480 on the private network: the metrics are open to anyone who can reach it.

Volumes and keys

Files are grouped into volumes. A volume is a directory that exists on two nodes, its replica set.

Every stored file gets a key:

v65/19/7e/197eccba0db40feb812546205fb436a2.mp4
│   │  │  │                                │
│   │  │  └ random 128-bit ID (hex)        └ extension, optional
│   └──┴ fan-out: the ID's first four hex digits
└ volume ID

The path on disk follows from the key (volumes/v65/19/7e/197e....mp4), and the volume map says which nodes hold volume 65. Finding a file therefore takes no lookup in any index: the key is the address. Keys are never reused, and a stored file never changes; an edit is a new key.

Volume states:

Replica states are active, syncing (being filled by a repair or a move) and draining (the old copy of a move, still serving reads for two minutes).

The coordinator keeps every healthy node with free space in at least one writable volume, and creates new volumes on the nodes with the most free space. A write picks among the writable volumes at random, weighted by the free space of their fullest replica, and twice as likely when the volume has a replica on the writing node, so most writes and later reads stay local.

Why volumes are capped at 100 GB

A volume is the unit that is repaired and moved, so its size sets how fast both happen.

• Repair speed. After a node failure its volumes have one copy until repair finishes. Many capped volumes, spread over many node pairs, are copied in parallel from and to many disks. A few huge volumes would each copy from a single disk, stretching the single-copy window from hours to days.
• Balance. Data moves a whole volume at a time, so a usage gap smaller than one volume cannot be evened out.
• Short read-only windows. A volume takes no writes while it moves.
• Bounded indexes. Each volume's index stays small.

The volume map

The map is the cluster's only shared state. Per volume: its ID, its type (files or name shard), its state and its replicas. Per node: its address, its free space and its heartbeat.

In cluster mode it lives in a Raft group inside WarrenFS (hashicorp/raft, the log in bbolt under dataDir/raft), carried over the internal listener as an upgraded HTTP connection with the peer token. Three or five instances vote (one for a test cluster); every other instance is a non-voting member with a full copy. So every instance reads the map locally, and a new node is added by the leader, without a vote, as soon as it answers health checks.

Losing the voters' majority stops changes to the map: no new volumes, no repairs, no moves. It does not stop reads and writes. See Failures and repair.

In kubernetes mode the map is a cluster-scoped Volume custom resource per volume (volumes.warrenfs.gsid.nl), the heartbeats are Leases named <cluster>-node-<node>, and the coordinator holds the Lease <cluster>-coordinator.

Health

In cluster mode every instance asks every other one for its heartbeat every 2 seconds (/peer/v1/health). The answer says how old the heartbeat is, so an instance that still answers HTTP while its own loops hang still looks silent. Health depends on nothing outside WarrenFS.

A heartbeat carries the node's internal address, its free space and the size of every volume it holds. Two rules follow from it, and neither needs anyone to change the map:

• A volume whose replica's heartbeat is older than repair.readonlyAfter takes no writes, and takes them again as soon as the heartbeat is back.
• A new volume takes writes only once every replica lists it in its heartbeat, which proves each replica has seen it.

On disk

/var/lib/warrenfs/
├── volumes/
│   └── v65/
│       ├── index.db                                      the volume's index (bbolt)
│       └── 19/7e/197eccba0db40feb812546205fb436a2.mp4    a stored file, as a plain file
├── names/s0 ... s63/names.db                             the name shards (bbolt)
├── raft/                                                 cluster mode: Raft log and snapshots
├── cache/                                                copies of files held on other nodes
└── scrub.state                                           when the last scrub pass finished

Each file's metadata (key, size, SHA-256, content type, creation time) is in its volume's bbolt index and also on the file itself, as the xattr user.warrenfs.meta. The record travels with the file. The data directory therefore needs a filesystem with user xattrs: ext4 and XFS have them, and WarrenFS checks at startup.

bbolt gives memory-mapped reads in microseconds and one writer per database. With one database per volume, writes to different volumes run in parallel, and concurrent writes to one volume are batched into one commit.

The node-local cache

A file of up to cache.maxFileSize (16 MB) that an instance had to fetch from a peer is copied into the node's cache in the background, checked against its checksum (cache.maxSize, 10 GB, least recently used out first). A key never changes, so a cached copy is never stale; deleting a key evicts it on every node. Files on the node's own disk need no cache of their own: the Linux page cache keeps hot ones in memory.

Next: Writes, reads and deletes.