Start

Writes, reads and deletes

A write returns its key only after every replica of the volume has the complete file on disk, fsynced, with the same size and SHA-256. This page follows a write, a read and a delete through the cluster, and spells out exactly what each answer means, so you know what to build on.

The write path

  1. The app sends the file to the instance on its own node: POST /v1/files by key, or PUT /v1/names/<bucket>/<path> by path. The mount and the CSI driver do the same on close.
  2. The instance picks a writable volume: every replica active and healthy, under the size cap, chosen at random weighted by free space, and preferring volumes with a replica on this node. If there is none, the answer is 503.
  3. The body streams to all replicas at once, the local one directly and the other over the private network. Nothing is buffered in memory; a 4 GB video costs no 4 GB of RAM.
  4. Each replica writes a temp file in the volume, computing the SHA-256 while the bytes arrive, sets the metadata xattr, fsyncs the file, renames it into place, fsyncs the directory, and only then records the entry in the volume's index. A crash at any point never leaves an index entry without a complete file behind it.
  5. The instance compares the results. Every replica must report success, the same SHA-256, and a size equal to the bytes it sent.
  6. Only then does the app get 201 and the key.

What an acknowledgement means

Answer What you can rely on
201 Created The complete file is on every replica of its volume (two nodes), fsynced, with identical checksums. Either node can lose power now; nothing is lost.
503 with Retry-After: 1 Nothing. A replica failed or none was writable. Copies already made are deleted again, and the key was never handed out. Send the file again; the retry may well land in another volume.
507 Insufficient Storage A data disk is down to its reserve (volumes.reserve, 20 GB). Add capacity.
connection lost mid-upload Nothing was stored: the temp files are removed.

Because the body is streamed, not buffered, the client retries, not the instance. The mount and the CSI driver already retry for up to 30 seconds on 503 and on an unreachable instance, which also rides out a restart of the instance on the node. Your own HTTP clients should do the same.

A write by name stores the file exactly like this, then records the name in its name shard on every healthy node (every node holds every shard). It fails, and should be retried, unless they all confirm. A shard copy on a node that is down is skipped and catches up from its peers when the node is back, so one node down never makes directories read-only.

Reads

By key. The instance looks the key's volume up in its local copy of the map.

  1. If this node holds an active replica, the file is served from local disk, through Go's http.ServeContent: Range, HEAD, If-None-Match and If-Modified-Since all work.
  2. Otherwise, a copy in the node-local cache is served if there is one.
  3. Otherwise, the request goes to a replica on another node, healthy ones first and silent ones as a last resort, and the response (status, range and headers) is passed through unchanged. Files up to 16 MB are then copied into the node's cache in the background, checked against their checksum.

Every response for a stored file carries the same headers:

Header Value
ETag the file's SHA-256, quoted
Cache-Control public, max-age=31536000, immutable
Last-Modified when the file was stored
Content-Type as stored with the file

By name. The name is looked up in the node's own copy of its shard, with no network round trip, and the file is then read by key as above.

If the index lists a file whose bytes are missing on the local disk, the read goes to the peer instead and the problem is logged; the next scrub repairs the local copy.

Deletes

  • A delete writes a tombstone on every replica it can reach and removes the file. It succeeds (204) as long as one replica is reachable. If every replica was reached and none had the key, the answer is 404; if none could be reached, 503.
  • Each instance pulls its peers' tombstones every minute, re-reading the last five minutes each time to absorb clock differences. A replica that was down catches up on the deletes it missed when it returns.
  • Tombstones win. A deleted file does not come back, also when the delete reached a replica before the write did.
  • Tombstones are kept for deletes.tombstoneRetention (30 days), then purged.
  • The space is free at once: a deleted file is an unlinked file. There is no compaction, apart from bbolt's own.
  • Deleting a key evicts its cached copies on every node.

Consistency, plainly

  • A key is immutable. No reader ever sees half a file or an old version under a key, and no cache can be stale.
  • A name is last-writer-wins. Every change carries a version: the writer's clock, but always above the version it replaces. Overwriting a name points it at the new file and deletes the old one.
  • Read-your-writes across nodes. A name write returns only after every healthy node's shard copy has it, so a read on any healthy node sees it. The mount trusts names and attributes for its cache timeout (1 second by default), so another node's mount may see a change up to that much later.
  • One bucket per app. Two apps can never write the same path; two writers in one app to one path resolve by version.

Next: what happens when a node fails, in Failures and repair.