`615754ca0747`

rendered

docs/features/usbd.adoc7.7 KiBrendered source history

Fleet-usbd

For airgapped deployments/system updates for low-tech users with limited internet access, it is better to have something more manual than pusher.

fleet-usbd is a simple update daemon, which watches for a USB stick with the system update to be inserted into the machine, reads the manifest (usbd-update) from it, copies the closure from the USB stick, switches the current system generation, moves the manifest (usbd-update.done) and reboots.

The stick is discovered by volume label; the label is only a discovery hint, not a trust anchor. Detection is a systemd device unit on /dev/disk/by-label/<label> with the daemon service bound to it (WantedBy=), no udisks dependency.

Update flow

Stick inserted, device unit starts the service, stick is mounted.
Daemon decrypts the manifest, checks the system name matches this host.
If the manifest’s system is already the current generation - noop. Guards against reinsertion when the .done rename previously failed.
Closure is copied from the stick. Hash verification is part of the copy; copying already-present paths is a noop.
New generation is set as the boot default (boot, not switch), so the existing on-boot health-check rollback applies.
Manifest is renamed to usbd-update.done.
Reboot.

Requirements

One file per store path: it should be possible to rewrite the system update on USB stick in incremental fashion: unchanged store paths keep their exact file name and bytes, only added/removed paths change on the stick.
Deterministic, non-leaking names and ciphertext: file names and file contents must be byte-stable across rewrites (otherwise incremental sync degrades to full rewrite), but must not leak store path names/contents. Mix a secret into the derivation: per-file key and file name are derived from (deployment secret, narHash); the deployment secret is an ordinary fleet shared secret owned by the target host. Derivation must use narHash, not the store path: input-addressed paths can be rebuilt nondeterministically, yielding the same store path with different contents, and a same-name-different-bytes file breaks both immutability and size-only synchronization. Note age cannot be used for the path files - its encryption is randomized, so ciphertext would churn on every rewrite.
Manifest encrypted to the host ssh key: the manifest is encrypted the same way fleet already performs secret encryption (age, ssh-ed25519 host key recipient). It carries the system name and the per-path metadata listed in the copying section.
Manifest should include system name: update daemon should check if the system update is suitable for it.
User feedback via callbacks: how the user learns "update running / done / safe to remove" differs per client (screen, beep, LED). The daemon only exposes callback hooks for overriding; no built-in feedback.
Backwards-compatible updates: an old usbd must be able to apply updates produced by a newer fleet (forward compatibility from the daemon’s point of view). Consequences: the on-stick format core (discovery label, manifest name and encryption, name/key derivation, file cipher, narinfo fields) is frozen from v1; the manifest is self-describing and old daemons ignore unknown fields, so evolution is additive-only. A truly breaking format change is still possible in two hops: every update ships the whole system including usbd itself, so the writer can emit the old format once to upgrade the daemon, then switch.

Copying store paths

The daemon exposes the stick as a standard nix binary cache on localhost and lets nix substitution perform the copy. This reuses nix machinery for hash verification, signature checking and skipping already-present paths - no diffing logic in the daemon.

Manifest carries, per store path: store path, narHash, narSize, references, build-time signature, on-stick file name and per-file key - everything needed to synthesize a narinfo.
Daemon binds an ephemeral HTTP listener on 127.0.0.1 serving the binary cache protocol: /nix-cache-info, synthesized <hash>.narinfo (Sig: from build time, URL: nar/<derived-name>, Compression: zstd) and /nar/<derived-name>, which streaming-decrypts the stick file (chunked AEAD) into the .nar.zst; nix does the decompression and NAR hash check itself.
The copy goes through the nix-eval FFI (Store::open on the localhost cache, copy_to into the local store), not a subprocess. A random token path prefix in the cache URL keeps other local processes from pulling decrypted NARs off the listener.
Signatures are verified by nix against trusted-public-keys; the deployer signing key is pinned there by the fleet module.
The copy holds a temporary GC root on the system path so a concurrent GC cannot race it. The listener is shut down after the copy, then the normal flow continues: set boot generation, rename manifest, reboot.

Stick writing is the mirror image and also goes through the FFI: query path infos of the system closure, sign, dump each path to NAR, zstd-compress, encrypt with the derived per-file key, write under the derived name, write the age-encrypted manifest last.

Stick format

FAT32 (mkfs.vfat): universally supported and more robust against unclean ejects than exFAT. The 4 GiB file size limit is handled by splitting large encrypted files into fixed-size chunks; chunk names are derived from (deployment secret, narHash, chunk index) and the manifest lists the chunk sequence per path. This is plain size-splitting, unrelated to dedup chunking. Filesystem is case-insensitive; derived file names should be lowercase.

Synchronization

Stick content should be synchronizable from http with a known tool that works on Windows. Flat content-addressed layout plus one manifest file satisfies this: rclone handles it (single portable exe, can be pre-installed on the stick itself with a .bat wrapper running rclone sync --progress; rclone has no native GUI and its experimental web GUI downloads assets from github on first run, so it is unusable here). Files are immutable under their derived names, so size-only comparison suffices. Sync should use --delete-before: rclone does not order transfers and by default deletes extraneous files last, which would require the stick to hold the old and new closure simultaneously. A partially synced stick is safe: the copy is hash-verified and the closure must be complete before the switch happens; the daemon should stat all manifest-referenced files upfront to report a partially synced stick cleanly.

Usage

On the host, enable the daemon and trust the update signing key:

{
  services.fleet-usbd = {
    enable = true;
    signPublicKeys = [ "client-1:..." ];
    # Optional user feedback script: receives copying/switching/done/noop/failed
    # hook = ./usbd-hook.sh;
  };
}

The daemon is bound to the volume label (FLEETUSBD by default) via a systemd device unit; inserting the stick runs fsck and applies the update.

Writing the stick (expects an inserted stick with the matching label, mounted via udisks):

mkfs.vfat -n FLEETUSBD /dev/sdX1
fleet usbd write HOSTNAME --sign-key ./client-1.secret

The signing keypair is generated once with nix key generate-secret --key-name client-1.

Not a requirement

Rollback prevention: not a concern for any of the current clients.
Update authenticity in the daemon: handled by nix itself via store signatures during the copy; nothing to do on the usbd level.
Chunking of large paths: attic-style sub-path chunking would help large paths that change slightly, but conflicts with ease of synchronization; skipped.
Status/log back-channel on the stick: postponed, depends on the client.
GC of old generations: not a daemon problem.
Multi-host sticks: conflicts with incremental stick rewrite; one stick per host for now.