Start
Replicated file storage for clusters you run yourself

Two copies of every file.
On the disks you already have.

WarrenFS turns the local disks of your nodes into one replicated store. A write is acknowledged only once it sits on two machines, every file stays a plain file on ext4 or XFS, and you grow by adding a node. One Go binary, with or without Kubernetes.
Start in five minutes How a write works
In production: a multi-tenant CMS moved onto it with zero downtime.
A write streams to both replicas at once. Your app gets its key only after both have fsynced the file and agree on its size and SHA-256.
2
copies on different nodes before every write returns
0
raw block devices: your data is files in a directory
2 s
between health checks, instance to instance, no outside arbiter
1
static binary: storage, Raft, repair, the mount and the CSI driver
The problem

Shared storage is where small clusters break.

Your apps scale across nodes. Their files usually do not. Uploads, media, themes and templates end up somewhere every pod can reach, and that somewhere quietly becomes the weakest part of the platform.

  1. One NFS server

    Every pod mounts the same box. When it reboots, everything waits; when its disks fill, you buy a bigger box. A single point of failure that also sets your capacity ceiling.

  2. A storage platform built for someone else

    The heavyweight options are robust, and they want raw block devices, a fleet of daemons and someone on call who knows them. That is a team, not a weekend.

  3. Files only the tool can read

    Stores that keep data as chunks in their own format are fine until the tool is what broke. Then your files are fragments on a disk.
The engineering case, and the options we ruled out
What WarrenFS does

A narrow tool that does one job properly.

Files are written whole and never changed in place. That one rule removes most of the hard problems of distributed storage, and puts the effort where it belongs: in what happens when things fail.

  • Two copies before the ack

    A write streams to both replicas of a volume at once. Each one checksums while it writes, fsyncs and renames into place. Your app gets its key only when both agree.

  • Plain files on your disks

    No raw devices, no chunk format. A file is a file in a directory on ext4 or XFS, its metadata in xattrs beside it, so an index can be rebuilt by scanning the disk.

  • Writable during an outage

    A volume with a silent replica turns read-only and new writes go to volumes whose replicas are all healthy. Two copies, and a node down does not stop uploads.

  • Heals itself

    A node silent for 15 minutes is replaced in every volume it held, refilled from the surviving copy and checked file by file. A weekly scrub re-reads every byte and repairs bad copies.

  • Grows by adding a node

    New volumes land on the emptiest nodes. When usage drifts apart the rebalancer moves whole volumes, one at a time, paced at 50 MB/s, never below two copies.

  • Mounts like a directory

    A FUSE mount and a CSI driver show a bucket as a folder. cp, tar, rsync, nginx and PHP's file_put_contents work unchanged.
Under the hood

What an acknowledgement means.

Storage is easy to judge on the happy path. WarrenFS is built around one promise instead: when your app holds a key, the file exists twice.

  1. Local first

    Your app sends the file to the WarrenFS instance on its own node, over the private network.

  2. A healthy volume

    The instance picks a writable volume: every replica healthy, room to spare, weighted by free space.

  3. Both replicas at once

    The bytes stream to both replicas at the same time. Each computes the SHA-256 while writing, fsyncs, and renames the file into place.

  4. Then, and only then, the key

    Both confirm the same size and checksum before your app gets its key. Anything less is a 503 with Retry-After, and the partial copies are tombstoned.
$ curl -s --data-binary @intro.mp4 \
    -H "Authorization: Bearer $WARRENFS_TOKEN" \
    -H "Content-Type: video/mp4" \
    "http://10.0.0.11:7480/v1/files?ext=mp4" | jq
{
  "key": "v12/9f/3a/9f3a6c0e5b7d4e21a8c3f0d9b6e2a1e1.mp4",
  "size": 48211934,
  "sha256": "5d41c0a9…e7b2",
  "contentType": "video/mp4"
}

# the key names the volume, never the node
$ curl -s -H "Range: bytes=0-1048575" \
    -H "Authorization: Bearer $WARRENFS_TOKEN" \
    http://10.0.0.11:7480/v1/files/v12/9f/3a/9f3a…e1.mp4 \
    -o head.mp4
Keys are immutable. An edit is a new write: your app stores the new file, switches its pointer and deletes the old key.
Failure is the normal case

A node just died. Here is the next quarter of an hour.

Nobody gets paged to move data around. Every step is automatic, and every timing is a setting.
  1. 0 s

    A node stops answering

    In cluster mode every instance checks every other one every 2 seconds. Health is decided inside WarrenFS, not by an outside service.
  2. 1 min

    Its volumes stop taking writes

    After repair.readonlyAfter, new writes go to volumes whose replicas are all healthy. Reads carry on from the other copy. Nobody changes anything.
  3. 15 min

    It is replaced

    After repair.repairAfter, another node joins every volume and name shard it held, copies from the survivor at full speed, and checks every file before it goes active.
  4. Later

    It comes back

    The returning node catches up on the deletes it missed, and drops its old copies only once each volume has its full set of active copies elsewhere.
  5. Weekly

    Every byte re-read

    Each node re-reads every file it holds against its checksum, at a capped rate. A missing or corrupt copy is replaced from a healthy replica.
Every failure case, including the ones two copies cannot cover
Two ways in

Keys for apps that track their files. Paths for apps that do not.

Use the HTTP API by key and keep keys in your own database, or store files under paths in a bucket and mount it like a directory. Both get the same two copies.
# a bucket as a directory, through FUSE
$ export WARRENFS_TOKEN=…
$ warrenfs mount --url http://10.0.0.11:7480 \
    --bucket shop /srv/shop &

# whole-file writes, stored with two copies on close
$ rsync -a uploads/ /srv/shop/uploads/
$ mv /srv/shop/uploads/new /srv/shop/uploads/banners
$ ls /srv/shop/uploads/banners | head -2
a1f0-banner.webp
a1f0-banner@2x.webp
# an inline CSI volume; the token stays in the app's namespace
volumes:
  - name: shop-files
    csi:
      driver: warrenfs
      volumeAttributes: {bucket: shop}
      nodePublishSecretRef: {name: warrenfs-shop}
containers:
  - name: shop
    volumeMounts:
      - name: shop-files
        mountPath: /srv/shop
        mountPropagation: HostToContainer
  • Range requests on every read, so video seeks without a local copy
  • One token and one bucket per app, revocable on its own
  • Mounts survive a restart of the storage instance and of the CSI plugin
Deploy

Runs where you run.

The volume map lives in a Raft group inside WarrenFS itself, so the same binary runs in Kubernetes, on bare metal or on your laptop, with no outside coordinator to run beside it.

  • Kubernetes

    A DaemonSet with host networking and a hostPath data directory, plus the CSI driver for pods. Your API server holds no storage state.

  • Bare metal

    The same binary as a systemd service. Machines mount buckets with warrenfs mount. No etcd, no external coordinator.

  • Your laptop

    Standalone mode: one instance, one copy, the same API. Build it, run it and store a file in five minutes.
Already on kubernetes mode? Moving the volume map from custom resources into Raft is a rolling restart. No downtime, nothing copied but the map.
Day two

One command tells you where you stand.

warrenfs status counts every byte once and on disk, and tells you how much new data still fits with two copies and the reserve. The same numbers are on /metrics for Prometheus.
$ warrenfs status
warrenfs cluster "warrenfs": 4 of 4 nodes healthy, 2 copies of everything

NODE   STATUS  DISK    FREE    WARRENFS  CACHE   FILES  VOLUMES  SHARDS  SEEN
node1  ok      1.9 TB  1.4 TB  212.4 GB  2.1 GB  31877  7        64      1s ago
node2  ok      1.9 TB  1.4 TB  208.9 GB  1.8 GB  31102  7        64      1s ago
node3  ok      3.8 TB  3 TB    421.7 GB  2.6 GB  63559  13       64      0s ago
node4  ok      3.8 TB  3 TB    419 GB    0 B     63210  13       64      2s ago

used           631 GB  in 94874 files, counted once
used on disk   1.3 TB  all 2 copies and the indexes
free on disk   8.8 TB  on the healthy nodes
free for data  4.4 TB  new data that fits with 2 copies and 20 GB reserve per node

volumes: 20 for files (12 writable, 8 sealed), 64 name shards; 0 degraded, 0 replicas syncing

  • Drain, don't migrate

    warrenfs drain node3 moves its volumes and name shards off, one at a time. undrain stops it.

  • Mutual TLS between nodes

    Each instance issues itself a certificate from a cluster CA made by warrenfs peer-ca. Switching it on takes two rolling restarts.

  • Config that cannot lie

    Unknown keys are an error, every problem is reported at once, and warrenfs config prints the effective settings with secrets redacted.

  • Logs that name the app

    One token per app: logs say which app wrote or deleted a file, and one app can be cut off without touching the rest.
Straight answers

Built for files that are written once.

WarrenFS keeps a narrow scope on purpose. If your data is in the left column, it is a very good fit. If it is in the right one, keep that part somewhere else.

A very good fit

  • Media: images, video and audio, read with ranges
  • User uploads and form attachments
  • Themes, templates, CSS and JS that apps read from disk
  • Build artifacts, exports and generated files
  • Anything written whole and replaced, not edited

Use something else

  • Databases, SQLite files and logs: anything changed in place or appended to
  • An S3-compatible API: WarrenFS speaks its own small HTTP API, and POSIX through the mount
  • Erasure coding: two full copies cost twice the raw space, and in return every copy is a readable file
  • Replication between sites: copies live on nodes of one private network
In production

A production CMS moved onto WarrenFS without a second of downtime.

A multi-tenant CMS keeps its themes, media and form attachments on WarrenFS. Its pods rolled over onto it while a probe kept requesting files, and a final checksum pass went over every file of the old share.
956
probe requests during the switch
0
of them failed
0
files missing after the checksum pass
Documentation

Written for the people who will run it.

Replicated file storage on the disks your nodes already have: every file is on two machines before the write returns, kept as a plain file on ext4 or XFS, and the cluster grows by adding a node.

Start here

  • Documentation: where to start, WarrenFS in one minute, and its guarantees and limits at a glance.
  • Why WarrenFS: the problem it solves, the options it was measured against, and the trade-offs it makes on purpose.
  • Quickstart: one instance on your machine: upload, ranged reads, paths and a FUSE mount, in five minutes.

Concepts

  • Architecture: one binary per node, volumes and keys, the volume map in Raft, and what lives where on disk.
  • Writes, reads and deletes: what happens on the wire, and exactly what an acknowledged write guarantees.
  • Failures and repair: a node failure minute by minute, repair, scrub, and the cases two copies cannot cover.
  • Names and the mount: paths in a bucket, the FUSE mount and the CSI driver, and what they refuse.

Operations

  • Kubernetes: cluster mode as a DaemonSet, and buckets mounted into pods with the CSI driver.
  • Bare metal: the same binary under systemd, with mutual TLS between machines.
  • Day two: status, draining and adding nodes, rebalancing, alerts, TLS and upgrades.

Reference

  • Configuration: every setting, its default, its environment variable, and the secrets.
  • HTTP API: every route, header and status code of the internal API.
  • CLI: every warrenfs command and its flags.
  • Metrics: every Prometheus metric, and which ones to alert on.
Where to start, and the guarantees at a glance

Put the disks you already have to work.

Start a standalone instance on your laptop today. Grow it into a cluster when you are ready, without changing a line in your apps.
Read the quickstart Study the architecture