Why WarrenFS
WarrenFS exists because one NFS server held every file of a growing platform, and none of the distributed storage systems on the shortlist fit a small cluster with ordinary disks. This page is the engineering case: the problem, the requirements, what was considered, and the trade-offs made on purpose.
The problem
All media and site assets lived on one NFS share, tens of terabytes, mounted into pods as a hostPath. Two things follow from that:
- One server is a single point of failure. When it reboots, every pod that reads a template or serves an image waits for it.
- Its disks are the capacity ceiling. Growing means a bigger box, not another one.
Meanwhile every other node in the cluster had disks of its own and a network uplink of its own, and did nothing for storage.
What it has to do
- Two copies of every file, on different nodes. Losing a node loses nothing.
- Fast reads, big and small. Videos with range requests and seeking, and thousands of small templates, images and stylesheets.
- Grow by adding nodes. No forklift upgrade, no fixed pool size.
- Sit on the host filesystem. No raw block devices. Data is plain files in a directory on each node's existing ext4 or XFS.
- Write and delete, never modify. A file is never changed in place; an edit is a new write.
- Private traffic stays private. Writes, replication, repair and rebalancing run on the private network.
What was considered
Every option below was assessed for a cluster of a handful of nodes, ordinary disks already partly in use, and a small team. Your constraints may weigh them differently.
| Option | Why it did not fit |
|---|---|
| Ceph (Rook, RGW and CephFS) | The most robust of the lot, but it needs raw devices and is heavy to operate. |
| Garage | Sits on the host filesystem, but with two copies one node down blocks writes, and data is stored as chunks, not as files. |
| SeaweedFS | A large number of open issues at the time. |
| MinIO | The open source edition has been archived. |
| RustFS | Too young to trust with the only copies of anything. |
| GlusterFS | Development has stalled. |
| MooseFS | The master is a single point of failure in the community edition. |
| Versity S3 Gateway | S3 in front of one filesystem: no replication. |
Why a narrow scope makes it tractable
General-purpose distributed filesystems are hard because files change: in-place updates, appends, locks, partial writes and caches that go stale. WarrenFS removes those problems by refusing them. A stored file never changes, so:
- a key always means the same bytes, and every cache of it is valid forever;
- replication is "write the same file twice and compare checksums", not a consensus protocol per byte;
- repair is a file copy checked against a checksum.
The pattern is proven: MogileFS and Facebook's Haystack stored immutable blobs in volumes the same way. The engineering effort goes where storage systems actually fail: re-replication after a node is lost, scrubbing for bit rot, deletes that must not come back, and moving data without dropping below two copies.
It is one Go binary. The standard library covers HTTP, streaming, range and conditional requests (http.ServeContent) and checksums; Raft comes from hashicorp/raft and the indexes are bbolt.
Trade-offs, made on purpose
- Two full copies cost twice the raw space. There is no erasure coding. In exchange every copy is a complete file you can read with
cat, and a repair is a copy, not a reconstruction. - Usable space is at most half the total, and never more than all nodes except the largest can hold together.
warrenfs statuscomputes exactly that number as "free for data". - No in-place edits and no appends. Writing a file again stores a new copy. Through the mount,
>>is refused. - No S3 API. There are two native ways in: an HTTP API by key or by path, and a FUSE mount (with a CSI driver for Kubernetes). Both are small enough to read in an afternoon.
- Two copies, not three. A volume whose two replicas sit on two nodes that fail together is unavailable until one of them comes back. See Failures and repair.
- Not for databases, logs or SQLite. Anything that changes files in place belongs on a local disk or a block volume.
In production
A multi-tenant CMS runs on WarrenFS in production: themes, media and form attachments, about 4.5 GB in about 14,000 files. Its pods mount the bucket with the CSI driver, and nginx X-Accel-Redirect and Twig work on the mount unchanged.
- The switch had no downtime: 956 probe requests during the rollover, 0 failed.
- A final checksum pass afterwards found no file missing.
- There was no write freeze: a sync job copied the old share into WarrenFS every 15 seconds while the pods rolled over.
Next: try it yourself in the Quickstart, or read how it is built in Architecture.