Kubernetes

Run WarrenFS in cluster mode as a DaemonSet on every node, and mount buckets into pods with the CSI driver. In cluster mode WarrenFS keeps its own state in Raft, so the Kubernetes API is only the way it gets scheduled, not something it depends on at runtime. This is how the production cluster runs it.

What you deploy

Object	Purpose
Namespace `warrenfs`	everything below
ConfigMap `warrenfs-config`	`config.yaml`, the same on every node
Secret `warrenfs-secrets`	client tokens, peer token, and the cluster CA when `peerTLS` is on
DaemonSet `warrenfs`	one storage instance per node, on the host network
CSIDriver `warrenfs` and DaemonSet `warrenfs-csi`	the CSI node plugin, with the node driver registrar
a Secret per app, in the app's namespace	the app's token, key `token`, for its CSI volumes

The Volume custom resource definition and RBAC for Leases and Volumes are needed only in kubernetes mode, or while a cluster moves from kubernetes mode to cluster mode.

Configuration

# config.yaml, mounted at /etc/warrenfs/config.yaml
mode: cluster
cluster: warrenfs

# Every voter's address (port internal.port when left out). Other nodes may
# be left out: they find the cluster through these and join without a vote.
peers:
  node1: 10.0.0.1
  node2: 10.0.0.2
  node3: 10.0.0.3
voters: [node1, node2, node3]

dataDir: /var/lib/warrenfs
replication: 2

internal:
  port: 7480

repair:
  readonlyAfter: 15s
  repairAfter: 15m

Node identity comes from the downward API, secrets from the Secret; neither belongs in this file. Every key is in the configuration reference.

The storage DaemonSet

The parts that matter, as plain Kubernetes YAML:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: warrenfs
  namespace: warrenfs
spec:
  selector:
    matchLabels:
      app: warrenfs
  template:
    metadata:
      labels:
        app: warrenfs
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7480"
        prometheus.io/path: /metrics
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: warrenfs
          image: registry.example.com/warrenfs:0.10.0
          args: ["serve", "--config", "/etc/warrenfs/config.yaml"]
          ports:
            - name: internal
              containerPort: 7480
          livenessProbe:
            httpGet:
              path: /healthz
              port: 7480
          readinessProbe:
            httpGet:
              path: /healthz
              port: 7480
          env:
            - name: WARRENFS_CONFIG
              value: /etc/warrenfs/config.yaml
            - name: WARRENFS_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: WARRENFS_NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: WARRENFS_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: WARRENFS_CLIENT_TOKENS
              valueFrom:
                secretKeyRef:
                  name: warrenfs-secrets
                  key: WARRENFS_CLIENT_TOKENS
            - name: WARRENFS_PEER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: warrenfs-secrets
                  key: WARRENFS_PEER_TOKEN
          volumeMounts:
            - name: config
              mountPath: /etc/warrenfs
            - name: data
              mountPath: /var/lib/warrenfs
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "2"
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: warrenfs-config
        - name: data
          hostPath:
            path: /var/lib/warrenfs
            type: DirectoryOrCreate

Why it looks like this:

hostNetwork: true: the internal listener binds the node's private IP (status.hostIP), so apps reach the instance on their own node at a fixed address, and instances reach each other on the private network without a Service in between.
hostPath data: the files are plain files on the node's own disk, which is the point. Use a directory on ext4 or XFS; the instance refuses to start on a filesystem without user xattrs.
WARRENFS_CONFIG is set so the admin commands run in the pod (/warrenfs status, /warrenfs drain) find the config too.
Resources: the requests and limits are what production runs with; the hot path is disk to socket and needs little CPU.
The image is built from the repository's Dockerfile: a static binary at /warrenfs on distroless, running as root because the data directory is a root-owned hostPath.

With peerTLS, add WARRENFS_PEER_CA_CERT and WARRENFS_PEER_CA_KEY from the same Secret; see Day two.

Apps that use the HTTP API

An app talks to the instance on its own node over the private network:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: WARRENFS_URL
    value: http://$(NODE_IP):7480
  - name: WARRENFS_TOKEN
    valueFrom:
      secretKeyRef:
        name: myapp-warrenfs
        key: token

The token must be the app's entry in WARRENFS_CLIENT_TOKENS, and the app's name is its bucket.

The CSI driver

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: warrenfs
spec:
  attachRequired: false
  volumeLifecycleModes: ["Ephemeral"]
  fsGroupPolicy: None

The plugin DaemonSet, warrenfs-csi, runs two containers on the host network:

plugin: the warrenfs image with args: ["csi", "--endpoint", "unix:///csi/csi.sock"], privileged and as root (it mounts FUSE file systems and bind-mounts them into pods). Environment: WARRENFS_NODE_NAME from spec.nodeName, and WARRENFS_URL=http://$(NODE_IP):7480 with NODE_IP from status.hostIP. Host paths: /var/lib/kubelet/plugins/warrenfs at /csi (the socket), /var/lib/kubelet/pods at the same path with mountPropagation: Bidirectional, and /var/lib/warrenfs-csi (the record of mounted volumes, so a restarted plugin mounts them again, plus the spool for files being written).
registrar: registry.k8s.io/sig-storage/csi-node-driver-registrar, with --csi-address=/csi/csi.sock and --kubelet-registration-path=/var/lib/kubelet/plugins/warrenfs/csi.sock, the socket directory at /csi and /var/lib/kubelet/plugins_registry at /registration.

Production gives the plugin 50m CPU and 64Mi memory, limited to 1 CPU and 512Mi.

Pin the plugin's image separately from the storage image and upgrade it on purpose: it holds the pods' mounts. Storage instances can be upgraded with a rolling restart while every mount stays up.

Mounting a bucket in a pod

Put the app's token in its own namespace:

kubectl create secret generic warrenfs-shop --namespace shop --from-literal=token=<the shop token>

and ask for the bucket inline:

spec:
  volumes:
    - name: data
      csi:
        driver: warrenfs
        volumeAttributes:
          bucket: shop
          uid: "1000"
          gid: "1000"
          cacheTimeout: 30s
        nodePublishSecretRef:
          name: warrenfs-shop
  containers:
    - name: app
      volumeMounts:
        - name: data
          mountPath: /data
          mountPropagation: HostToContainer

mountPropagation: HostToContainer is what lets a remounted volume reach a running pod after the plugin restarts. Leave it out and a plugin restart leaves that pod with a dead mount until the pod restarts. The attributes are described in Names and the mount.

Check it

kubectl exec -n warrenfs <warrenfs pod> -- /warrenfs status

What the output means, and everything else you do after day one, is in Day two.