Start

Bare metal

The same binary runs without Kubernetes: one systemd service per machine in cluster mode, the volume map in Raft among the machines, mutual TLS between them, and warrenfs mount where apps need a directory. Nothing else to install: no database, no coordination service, no sidecar.

What you need

  • Three or more Linux machines on a private network, each with a data directory on ext4 or XFS (user xattrs).
  • Port 7480 open between the machines, and to nobody else: /metrics and /healthz answer without a token.
  • The warrenfs binary on every machine as /usr/local/bin/warrenfs:
go build -trimpath -ldflags "-s -w -X main.version=0.10.0" -o warrenfs ./cmd/warrenfs

1. Make the cluster CA

Once, on any machine:

warrenfs peer-ca /etc/warrenfs

wrote /etc/warrenfs/ca.key
wrote /etc/warrenfs/ca.crt

It never overwrites existing files, and writes the key with mode 0600. Copy both files to /etc/warrenfs/ on every machine. Each instance issues itself a certificate for its own name and IP from this CA, valid for a week and renewed daily; peers check each other's. Keep the key as secret as the peer token: whoever has it can join the cluster.

2. The config, the same everywhere

/etc/warrenfs/config.yaml:

mode: cluster
cluster: warrenfs
dataDir: /var/lib/warrenfs
replication: 2

# Every voter's address. Other machines join through any of these: a new one
# needs only this list and is added without a vote.
peers:
  node1: 10.0.0.1
  node2: 10.0.0.2
  node3: 10.0.0.3
voters: [node1, node2, node3]

internal:
  port: 7480

# Mutual TLS between the machines.
peerTLS: required

3. Identity and secrets, per machine

/etc/warrenfs/env, mode 0600:

WARRENFS_NODE_NAME=node1
WARRENFS_NODE_IP=10.0.0.1
WARRENFS_CLIENT_TOKENS=app=<token>
WARRENFS_PEER_TOKEN=<token>
WARRENFS_PEER_CA_CERT=/etc/warrenfs/ca.crt
WARRENFS_PEER_CA_KEY=/etc/warrenfs/ca.key

WARRENFS_PEER_TOKEN and the client tokens are the same on every machine; the name and IP are the machine's own. Generate tokens with something like openssl rand -hex 32.

4. The service

/etc/systemd/system/warrenfs.service (the repository's deploy/warrenfs.service):

[Unit]
Description=warrenfs storage
After=network-online.target local-fs.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/warrenfs serve --config /etc/warrenfs/config.yaml
EnvironmentFile=/etc/warrenfs/env
Restart=always
RestartSec=2
# Shutdown hands over Raft leadership and waits for running requests.
TimeoutStopSec=45
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now warrenfs

The voters form the Raft group on first start; every machine then checks every other one's health every 2 seconds.

5. Check it

The admin commands talk to the instance on the machine, with its config and peer token:

sudo bash -c 'set -a; . /etc/warrenfs/env; exec warrenfs status --config /etc/warrenfs/config.yaml'

The output is explained in Day two. Before the first writes, every node should be ok and the last lines should name a Raft leader.

Mount a bucket on a machine

Apps on a machine use the instance on that machine, through the HTTP API or a mount:

mkdir -p /srv/app-files
WARRENFS_TOKEN=<the app token> warrenfs mount --url http://10.0.0.1:7480 --bucket app \
    --uid 1000 --gid 1000 --allow-other /srv/app-files

The mount runs in the foreground until stopped, so run it as a service of its own, ordered after warrenfs.service. --allow-other lets users other than the mounting one see the files; a non-root mount needs user_allow_other in /etc/fuse.conf for that. Files being written are spooled in the system temp directory unless --spool says otherwise. What the mount does and refuses is in Names and the mount.

Add a machine

Give it the same config.yaml (its peers must name at least one running instance), its own name and IP in /etc/warrenfs/env, the CA files, and start the service. The Raft leader adds it as a non-voting member once it answers health checks. It fills its copy of the name shards, joins new volumes, and the rebalancer moves volumes to it when the usage gap calls for it; see Day two.