Start

Configuration

Every WarrenFS setting, its default and its environment variable, and the secrets, which come from the environment only. Settings are validated at startup and every problem is reported at once; an unknown key in the file is an error, so a typo never passes silently.

How values are resolved

  1. Built-in defaults.
  2. The YAML file: --config FILE, or $WARRENFS_CONFIG.
  3. Environment variables. Every key has one: WARRENFS_ plus the key path in upper snake case (volumes.maxSize is WARRENFS_VOLUMES_MAX_SIZE).

Secrets are read from the environment only, never from the file.

Two commands show the result:

  • warrenfs config prints the effective configuration, secrets redacted, and the client app names.
  • warrenfs env lists the environment variable for every key.

Sizes are written as 100GB, 1MB or 512KiB: KB, MB, GB and TB are powers of 1000, KiB, MiB, GiB and TiB powers of 1024. A plain number is bytes. Durations are Go durations: 90s, 15m, 720h.

Where each setting lives

What Where Takes effect
Static settings: ports, sizes, timeouts, replication the YAML file (in Kubernetes a ConfigMap, mounted at /etc/warrenfs/config.yaml) restart
Per-environment overrides WARRENFS_* environment variables restart
Node identity WARRENFS_NODE_NAME, WARRENFS_NODE_IP (in Kubernetes from the downward API: spec.nodeName, status.hostIP) restart
Secrets environment variables (in Kubernetes from a Secret) restart
Node drain cluster mode: warrenfs drain and undrain, kept in Raft; kubernetes mode: the node label warrenfs/drain=true live
Volume placement the volume map, written by the coordinator only live

What is the same everywhere and reviewed goes in git (the file, the environment, sealed secrets). What describes the running cluster (drains, where volumes are) lives in the volume map.

Cluster

Key Environment Default Meaning
mode WARRENFS_MODE kubernetes cluster: one instance per node, the volume map in Raft among them; in Kubernetes or on bare metal. kubernetes: the same, with the map, heartbeats and coordinator election in the Kubernetes API. standalone: one instance, one copy, for development.
cluster WARRENFS_CLUSTER warrenfs Name of the cluster. In kubernetes mode the prefix of the Lease names and the value of the warrenfs.gsid.nl/cluster label on the Volume resources.
namespace WARRENFS_NAMESPACE warrenfs Namespace of the Leases. Required in kubernetes mode and with migrateFromKubernetes.
peers WARRENFS_PEERS Cluster mode: node names and internal addresses, node1: 10.0.0.1 (port internal.port when left out). Must name every voter; other nodes may be left out, they are found through the peers. Environment form: node1=10.0.0.1,node2=10.0.0.2.
voters WARRENFS_VOTERS Cluster mode: the nodes that vote in Raft, an odd number: 3 or 5 (1 for a test cluster). Every voter needs an address in peers. The others follow with a full copy of the map and no vote. Environment form: node1,node2,node3.
peerTLS WARRENFS_PEER_TLS off Mutual TLS between instances: off, on or required. Each instance issues itself a certificate for its name and IP from the cluster CA and renews it daily. on dials TLS and falls back to plain for peers that do not speak it yet; required refuses peer requests without TLS. Apps keep plain HTTP to the instance on their node, on the same port. Not in standalone mode.
migrateFromKubernetes WARRENFS_MIGRATE_FROM_KUBERNETES false Cluster mode only: copy the volume map from the Volume resources once every instance runs in cluster mode. Can come out once warrenfs status shows volume map: raft.
replication WARRENFS_REPLICATION 2 Copies of every file. Must be 1 in standalone mode, at least 2 in the other modes.

Node and storage

Key Environment Default Meaning
node.name WARRENFS_NODE_NAME This node's name. Required in cluster and kubernetes mode.
node.ip WARRENFS_NODE_IP This node's private IP, where the other instances reach it. Required in cluster and kubernetes mode.
dataDir WARRENFS_DATA_DIR /var/lib/warrenfs Data directory on the host filesystem. Needs user xattrs (ext4, XFS); checked at startup.
logLevel WARRENFS_LOG_LEVEL info debug, info, warn or error. Logs are JSON on stderr.
internal.bind WARRENFS_INTERNAL_BIND node IP Listen address of the internal API. Empty means node.ip, or 127.0.0.1 when that is unknown.
internal.port WARRENFS_INTERNAL_PORT 7480 Port of the internal API, used by apps and by peers.
volumes.maxSize WARRENFS_VOLUMES_MAX_SIZE 100GB A volume is sealed for new writes at this size.
volumes.reserve WARRENFS_VOLUMES_RESERVE 20GB Free space kept on the data disk; writes stop below it (507).
cache.maxSize WARRENFS_CACHE_MAX_SIZE 10GB Disk space for node-local copies of files held on other nodes, under dataDir/cache. 0 turns the cache off.
cache.maxFileSize WARRENFS_CACHE_MAX_FILE_SIZE 16MB Largest file the cache keeps.

Repair, scrub, rebalancing, deletes

Key Environment Default Meaning
repair.readonlyAfter WARRENFS_REPAIR_READONLY_AFTER 1m A replica whose heartbeat is older than this: its volumes take no writes until it is back. At least 10s in cluster mode (heartbeats every 2 s; 15s is a good value), at least 30s in kubernetes mode.
repair.repairAfter WARRENFS_REPAIR_REPAIR_AFTER 15m A replica silent this long is replaced in every volume and name shard it held, filled from the surviving copy and checked. Must be longer than readonlyAfter.
scrub.every WARRENFS_SCRUB_EVERY 168h How often each node re-reads every file it holds and checks its checksum. 0 turns scrubbing off. A node's first pass starts an hour after it comes up.
scrub.maxRate WARRENFS_SCRUB_MAX_RATE 50MB Most bytes a second a scrub reads, per node.
rebalance.enabled WARRENFS_REBALANCE_ENABLED true Automatic volume moves: off drained nodes, and from the fullest node to the emptiest. Drains need it on.
rebalance.spreadThreshold WARRENFS_REBALANCE_SPREAD_THRESHOLD 10 Disk usage gap, in percentage points, that triggers moves. Between 0 and 100.
rebalance.holdTime WARRENFS_REBALANCE_HOLD_TIME 30m How long the gap must last before moves start.
rebalance.maxRate WARRENFS_REBALANCE_MAX_RATE 50MB Most bytes a second a move copies. Repairs copy at full speed.
deletes.tombstoneRetention WARRENFS_DELETES_TOMBSTONE_RETENTION 720h How long tombstones are kept before they are purged.

Public read path (not built yet)

These keys exist and are validated when public.enabled is true, but no public listener is built yet. Leave public.enabled off.

Key Environment Default Meaning
public.enabled WARRENFS_PUBLIC_ENABLED false The public read-only listener.
public.port WARRENFS_PUBLIC_PORT 7481 Its port, behind an ingress. Must differ from internal.port.
public.entryHost WARRENFS_PUBLIC_ENTRY_HOST Stable public hostname.
public.nodeHost WARRENFS_PUBLIC_NODE_HOST Per-node public hostname; must contain {n} (the number at the end of the node name, node3 gives 3) or {node} (the full name).
public.redirectThreshold WARRENFS_PUBLIC_REDIRECT_THRESHOLD 1MB Larger files would be redirected to a node that holds them.

Secrets

Variable Meaning
WARRENFS_CLIENT_TOKENS name=token pairs, comma separated: shop=<token>,media=<token>. One token per app; the name is also the app's bucket. At least one is required, and a token may not be used twice.
WARRENFS_PEER_TOKEN Authenticates traffic between instances, Raft included. Required in cluster and kubernetes mode.
WARRENFS_PEER_CA_CERT, WARRENFS_PEER_CA_KEY The cluster CA for peerTLS, as PEM or as the path of a PEM file. warrenfs peer-ca DIR makes one. Required when peerTLS is on or required. Whoever has the key can join the cluster.
WARRENFS_URL_SIGNING_KEY HMAC key for the signed URLs of the public read path. Required when public.enabled.

Client apps

Apps read their own settings, not WarrenFS's:

Variable Meaning
WARRENFS_URL The instance on the app's node: http://$(NODE_IP):7480, with NODE_IP from status.hostIP. Also the default --url of warrenfs mount and warrenfs csi.
WARRENFS_TOKEN The app's token, matching its entry in WARRENFS_CLIENT_TOKENS. Read by warrenfs mount.

Examples

Standalone, for development:

mode: standalone
replication: 1
dataDir: ./data
logLevel: debug
internal:
  bind: 127.0.0.1
  port: 7480
volumes:
  maxSize: 1GB
  reserve: 1GB

Cluster mode, the same file on every node; identity and secrets from the environment:

mode: cluster
cluster: warrenfs
dataDir: /var/lib/warrenfs
replication: 2
peers:
  node1: 10.0.0.1
  node2: 10.0.0.2
  node3: 10.0.0.3
voters: [node1, node2, node3]
internal:
  port: 7480
peerTLS: required
repair:
  readonlyAfter: 15s
  repairAfter: 15m