Names and the mount

Apps that keep their own file records store files by key. Apps written for plain files get paths instead: a bucket per app, real directories, rename, and a FUSE mount that makes the bucket an ordinary directory for nginx, PHP, Twig, cp, tar and rsync. In Kubernetes, the CSI driver puts that mount into a pod.

Buckets

A name is <bucket>/<path>. The bucket is the app's name in WARRENFS_CLIENT_TOKENS: with shop=<token>, the shop token reaches shop/... and nothing else; any other bucket answers 403. One token per app means access can be revoked per app, and the logs show which app wrote or deleted what.

Paths are at most 4096 bytes, each element at most 255 bytes; empty elements, ., .. and NUL are refused.

Entries and shards

A name is an entry pointing at a stored file. The file itself is stored like any other (two copies, a volume, a key); the entry holds the key, size, SHA-256, content type, modification time and a version. Directories are entries of their own, so empty directories exist and mkdir works.

The name index is split into 64 shards, each a bbolt database, and every node holds a copy of every shard. An entry lives in the shard chosen by a hash of its parent directory, so:

• a lookup never leaves the node;
• listing a directory reads one shard;
• names survive any number of node failures short of the last one.

Names are small and read far more often than changed, which is what makes a copy on every node cheap. A node that joins gets its shard copies filled like a repair.

Operations

A rename changes names, not files: the key stays the same. Renaming a directory moves its entries one by one, so it is not atomic: a failure halfway leaves part of the tree under each name, and running the same rename again finishes it.

Every change carries a version (the writer's clock, always above the version it replaces) and the newest wins. Deleted names leave tombstones that the shard copies pull from each other, like deleted files. The exact routes are in the HTTP API reference.

The FUSE mount

WARRENFS_TOKEN=<token> warrenfs mount --url http://<node IP>:7480 --bucket <bucket> /mnt/<bucket>

The mount talks to the WarrenFS instance on the same machine, by name for lookups and by key for reads.

What works

• Reads, fast: a file up to 1 MiB is fetched whole on its first read; a larger one in ranges that grow from 128 KiB to 8 MiB while it is read sequentially. An inode stands for one key, and a key never changes, so the kernel keeps the file's pages across opens.
• Whole-file writes: create or truncate, write, close. The bytes collect in a local spool file and are stored, with both copies, on close (or fsync). A failed store is returned by close, where programs that check for errors will see it.
• Directories: mkdir, rmdir, rename (also with RENAME_NOREPLACE), unlink, listing.
• Modification times: touch and utimes are kept.
• In practice: >, cp, tar, rsync, PHP's file_put_contents, nginx X-Accel-Redirect and Twig template loading all work unchanged.

What costs more than it looks

• Writing into the middle of an existing file works, but downloads the whole file first and stores it as a new file on close.

What is refused or ignored

df on the mount shows a fixed 1 TiB: the space is the cluster's, not one disk's. The real numbers are in warrenfs status. Names and attributes are trusted for the cache timeout (1 second; the CSI driver's cacheTimeout), so another node's change shows up at most that much later.

If the local instance restarts, requests wait and retry for up to 30 seconds instead of failing.

The CSI driver

warrenfs csi is a CSI node plugin named warrenfs: no controller, inline ephemeral volumes only. A pod asks for a bucket directly in its spec, with the token in a Secret in its own namespace:

volumes:
  - name: data
    csi:
      driver: warrenfs
      volumeAttributes:
        bucket: shop
        uid: "1000"
        gid: "1000"
        cacheTimeout: 30s
      nodePublishSecretRef:
        name: warrenfs-shop
containers:
  - name: app
    volumeMounts:
      - name: data
        mountPath: /data
        mountPropagation: HostToContainer

The plugin runs as its own DaemonSet, separate from the storage instances, so restarting or upgrading storage does not touch the mounts. When the plugin itself restarts it mounts every volume it served again; the pod must mount the volume with mountPropagation: HostToContainer for the new mount to reach it. Deploying the plugin is in Kubernetes.

What not to put on it

• Files that change in place: databases, SQLite, logs, anything appended to. Use a local disk or a block volume.
• Scratch space whose bytes need no second copy. It works, it just costs two disks and a network hop.

Files that are written whole and read many times are the sweet spot: media, uploads, themes and templates, generated derivatives, build artifacts. Image derivative caches and upload temp files fit too, when the app writes them whole: a derivative made on one node then serves every node.