Metrics

Every WarrenFS instance serves Prometheus metrics at /metrics on its internal port, without a token: the cluster's space, nodes and volumes, the instance's own requests and scrub, and in cluster mode its view of Raft. Grafana needs nothing else.

Two kinds of metric:

Cluster-wide: computed by every instance from its own copy of the volume map, so each instance reports the same numbers. Aggregate with max, not sum.
Per instance: requests, scrub and Raft describe the instance that serves them. Aggregate with sum or keep them per instance.

In Kubernetes the pods carry the usual prometheus.io/scrape, prometheus.io/port: "7480" and prometheus.io/path: /metrics annotations; see Kubernetes.

Space

Cluster-wide gauges.

Metric	Meaning
`warrenfs_used_bytes`	live bytes stored, every file counted once
`warrenfs_files`	live files stored
`warrenfs_used_on_disk_bytes`	bytes WarrenFS takes on all disks: every copy plus the indexes
`warrenfs_free_bytes`	free space on the disks of the healthy nodes
`warrenfs_free_for_data_bytes`	new data that fits with the configured number of copies and the reserve

Nodes and volumes

Cluster-wide gauges.

Metric	Labels	Meaning
`warrenfs_nodes`	`state`: `healthy`, `silent`	nodes by heartbeat state
`warrenfs_volumes`	`type`: `files`, `names`; `state`: `writable`, `sealed`, `other`, `any`	volumes by type and state; `other` is read-only for now (a silent or syncing replica), name shards are counted as `type="names",state="any"`
`warrenfs_degraded_volumes`		volumes and name shards with fewer active copies than configured
`warrenfs_syncing_replicas`		replicas being filled by a repair or a move

Per node, labelled node:

Metric	Meaning
`warrenfs_node_healthy`	1 when the node's heartbeat is recent
`warrenfs_node_heartbeat_age_seconds`	age of the node's last heartbeat
`warrenfs_node_disk_capacity_bytes`	size of the node's data disk
`warrenfs_node_disk_free_bytes`	free space on the node's data disk
`warrenfs_node_stored_bytes`	bytes WarrenFS holds on the node: file copies and indexes
`warrenfs_node_cache_bytes`	the node's cached copies of files held on other nodes
`warrenfs_node_files`	file copies on the node
`warrenfs_node_volumes`	volume copies on the node
`warrenfs_node_name_shards`	name shard copies on the node

Requests

Per-instance counters.

Metric	Labels	Meaning
`warrenfs_requests_total`	`op`, `code`	API requests served by this instance
`warrenfs_request_bytes_total`	`direction`: `in`, `out`	bytes received and sent by this instance's API

op is the API and the method, without the key or path: files_post, files_get, files_head, files_delete, names_put, names_get, names_post, names_delete, peer_get, peer_post, metrics_get, healthz_get, and other_<method> for anything else. code is the status class: 2xx, 3xx, 4xx, 5xx. Peer traffic (replication, repair, health checks) is counted under peer_*.

Scrub

Per instance.

Metric	Type	Meaning
`warrenfs_scrub_files_total`	counter	files this instance's scrub checked
`warrenfs_scrub_bytes_total`	counter	bytes this instance's scrub read and found good
`warrenfs_scrub_bad_total`	counter	missing or corrupt copies the scrub found
`warrenfs_scrub_repaired_total`	counter	bad copies the scrub replaced from a peer
`warrenfs_scrub_failed_total`	counter	bad copies the scrub could not repair
`warrenfs_scrub_last_pass_timestamp_seconds`	gauge	when this instance last finished a full scrub (0: never)
`warrenfs_scrub_running`	gauge	1 while a scrub pass runs

Raft

Cluster mode only, per instance, gauges.

Metric	Meaning
`warrenfs_raft_leader`	1 when this instance is the Raft leader, and so the coordinator
`warrenfs_raft_has_leader`	1 when this instance knows a Raft leader: the volume map can change
`warrenfs_raft_members`	servers in the Raft configuration, as this instance sees it
`warrenfs_raft_voters`	voters in the Raft configuration, as this instance sees it
`warrenfs_raft_applied_index`	last Raft log entry this instance applied to its volume map
`warrenfs_volume_map_in_raft`	1 once the volume map lives in Raft rather than in Kubernetes

warrenfs_raft_has_leader at 0 on every instance means the voters lost their majority. Exactly one instance should report warrenfs_raft_leader 1.

Other

Metric	Meaning
`warrenfs_scrape_time_seconds`	Unix time of this scrape

Queries worth keeping

# Data stored, and room left for new data
max(warrenfs_used_bytes)
max(warrenfs_free_for_data_bytes)

# Disk usage per node, in percent
100 * (1 - max by (node) (warrenfs_node_disk_free_bytes) / max by (node) (warrenfs_node_disk_capacity_bytes))

# Write and read rate across the cluster
sum(rate(warrenfs_requests_total{op=~"files_post|names_put"}[5m]))
sum(rate(warrenfs_request_bytes_total{direction="out"}[5m]))

# Server errors, by operation
sum by (op) (rate(warrenfs_requests_total{code="5xx"}[5m]))

The alerts that matter are listed in Day two.