1== Fleet-usbd23For airgapped deployments/system updates for low-tech users with limited internet access, it is better to have something more manual than pusher.45fleet-usbd is a simple update daemon, which watches for a USB stick with the system update to be inserted into the machine, finds its manifest on it, copies the closure from the USB stick, switches the current system generation, renames the manifest to *.done and reboots.67The stick is discovered by volume label; the label is only a discovery hint, not a trust anchor. Detection is a systemd device unit on /dev/disk/by-label/<label> with the daemon service bound to it (WantedBy=), no udisks dependency.89== Update flow10111. Stick inserted, device unit starts the service, stick is mounted.122. Daemon scans usbd-update-* files and tries to decrypt each; the one encrypted to this host's key is its manifest (the daemon has no naming secret, so it cannot derive its manifest name directly). The system name inside is checked as well.133. If the manifest's system is already the current generation - noop. Guards against reinsertion when the .done rename previously failed.144. Closure is copied from the stick. Hash verification is part of the copy; copying already-present paths is a noop.155. New generation is set as the boot default (boot, not switch), so the existing on-boot health-check rollback applies.166. Manifest is renamed to *.done.177. Reboot.1819== Requirements2021One file per store path:: it should be possible to rewrite the system update on USB stick in incremental fashion: unchanged store paths keep their exact file name and bytes, only added/removed paths change on the stick.22Deterministic, non-leaking names and ciphertext:: file names and file contents must be byte-stable across rewrites (otherwise incremental sync degrades to full rewrite), but must not leak store path names/contents. Mix a secret into the derivation: per-file key and file name are derived from (naming secret, narHash); the naming secret is project-wide, writer-side state in the file referenced by the `usbd.namingSecretFile` option (relative to the project directory, generated on first write). Hosts never need it - each manifest carries the keys for its paths - and because it is shared across hosts, a store path used by two hosts maps to the same file name and key, so shared sticks dedup naturally. Derivation must use narHash, not the store path: input-addressed paths can be rebuilt nondeterministically, yielding the same store path with different contents, and a same-name-different-bytes file breaks both immutability and size-only synchronization. Note age cannot be used for the path files - its encryption is randomized, so ciphertext would churn on every rewrite.23Manifest encrypted to the host ssh key:: the manifest is encrypted the same way fleet already performs secret encryption (age, ssh-ed25519 host key recipient). It carries the system name and the per-path metadata listed in the copying section.24Manifest should include system name:: update daemon should check if the system update is suitable for it.25User feedback via callbacks:: how the user learns "update running / done / safe to remove" differs per client (screen, beep, LED). The daemon only exposes callback hooks for overriding; no built-in feedback.26Backwards-compatible updates:: an old usbd must be able to apply updates produced by a newer fleet (forward compatibility from the daemon's point of view). Consequences: the on-stick format core (discovery label, manifest name and encryption, name/key derivation, file cipher, narinfo fields) is frozen from v1; the manifest is self-describing and old daemons ignore unknown fields, so evolution is additive-only. A truly breaking format change is still possible in two hops: every update ships the whole system including usbd itself, so the writer can emit the old format once to upgrade the daemon, then switch.2728== Copying store paths2930The daemon exposes the stick as a standard nix binary cache on localhost and lets nix substitution perform the copy. This reuses nix machinery for hash verification, signature checking and skipping already-present paths - no diffing logic in the daemon.31321. Manifest carries, per store path: store path, narHash, narSize, references, build-time signature, on-stick file name and per-file key - everything needed to synthesize a narinfo.332. Daemon binds an ephemeral HTTP listener on 127.0.0.1 serving the binary cache protocol: /nix-cache-info, synthesized <hash>.narinfo (Sig: from build time, URL: nar/<derived-name>, Compression: zstd) and /nar/<derived-name>, which streaming-decrypts the stick file (chunked AEAD) into the .nar.zst; nix does the decompression and NAR hash check itself.343. The copy goes through the nix-eval FFI (Store::open on the localhost cache, copy_to into the local store), not a subprocess. A random token path prefix in the cache URL keeps other local processes from pulling decrypted NARs off the listener.354. Signatures are verified by nix against trusted-public-keys; the deployer signing key is pinned there by the fleet module.365. The daemon holds a temporary GC root on every update path for the duration of the copy (temp roots protect not-yet-valid paths individually, so intermediate paths are covered before the toplevel becomes reachable), so a concurrent GC cannot race it. The listener is shut down after the copy, then the normal flow continues: set boot generation, rename manifest, reboot.3738Stick writing is the mirror image and also goes through the FFI: query path infos of the system closure, sign, dump each path to NAR, zstd-compress, encrypt with the derived per-file key, write under the derived name, write the age-encrypted manifest last.3940== Stick format4142FAT32 (mkfs.vfat): universally supported and more robust against unclean ejects than exFAT. The 4 GiB file size limit is handled by splitting large encrypted files into fixed-size chunks; chunk names are derived from (deployment secret, narHash, chunk index) and the manifest lists the chunk sequence per path. This is plain size-splitting, unrelated to dedup chunking. Filesystem is case-insensitive; derived file names should be lowercase.4344Layout:4546/usbd-update-<derive(hostname)>:: age-encrypted manifest for one host, renamed to *.done after a successful update. One manifest per host, so several hosts can share a stick.47/update-data/<xx>/<name>:: encrypted path chunks under derived names, sharded into 256 subdirectories by the first two name characters.48/gc/<derive(hostname)>:: plaintext list of chunk names referenced by that host's update (bare names, the shard prefix is implied). The writer refreshes its own list, then deletes only data files referenced by no list, so several hosts' updates can share one stick without the writers deleting each other's files.49/logs/<timestamp>-<random>.export.gpg:: journal dump in `journalctl -o export` format, written by the daemon after every run when `logRecipientFile` is set. Encrypted with OpenPGP to the operator key, so it can be decrypted with a smartcard. The random suffix avoids collisions between hosts sharing a stick (the daemon cannot derive its host id, and a plaintext hostname would leak).5051FAT32 allows at most 65536 directory entries per directory; a 32-character name costs 4 entries (3 LFN + 1 short), so a flat directory would cap out at ~16000 files. The 256-way shard gives each subdirectory its own budget (~4M files total), and as a side effect keeps the volume root nearly empty, so even a FAT16-formatted stick with its fixed 512-entry root directory works.5253== Synchronization5455Stick content should be synchronizable from http with a known tool that works on Windows. Flat content-addressed layout plus one manifest file satisfies this: rclone handles it (single portable exe, can be pre-installed on the stick itself with a .bat wrapper running `rclone sync --progress`; rclone has no native GUI and its experimental web GUI downloads assets from github on first run, so it is unusable here). Files are immutable under their derived names, so size-only comparison suffices. Sync should use --delete-before: rclone does not order transfers and by default deletes extraneous files last, which would require the stick to hold the old and new closure simultaneously. A partially synced stick is safe: the copy is hash-verified and the closure must be complete before the switch happens; the daemon should stat all manifest-referenced files upfront to report a partially synced stick cleanly.5657== Usage5859On the host, enable the daemon and trust the update signing key:6061[source,nix]62----63{64 services.fleet-usbd = {65 enable = true;66 signPublicKeys = [ "client-1:..." ];67 68 69 70 71 72 73 74 75 76 77 };78}79----8081The daemon is bound to the volume label (FLEETUSBD by default) via a systemd device unit; inserting the stick runs fsck and applies the update.8283Writing the stick (expects an inserted stick with the matching label, mounted via udisks):8485[source,shell]86----87mkfs.vfat -n FLEETUSBD /dev/sdX188fleet usbd write HOSTNAME89----9091The signing keypair is generated once with `nix key generate-secret --key-name client-1`; the secret key file is referenced from the fleet configuration (or overridden with `--sign-key`):9293[source,nix]94----95{96 hosts.HOSTNAME.usbd.signingKeyFiles = [ "client-1.secret" ];97}98----99100Paths are resolved relative to the fleet project directory and deliberately are not nix paths, so the secret key is never imported into the store.101102Reading a journal dump from the stick:103104[source,shell]105----106gpg -d logs/2026-07-07-10-10-20-1a2b.export.gpg | systemd-journal-remote -o /tmp/metis.journal -107journalctl --file /tmp/metis.journal -b108----109110== Not a requirement111112Rollback prevention:: not a concern for any of the current clients.113Update authenticity in the daemon:: handled by nix itself via store signatures during the copy; nothing to do on the usbd level.114Chunking of large paths:: attic-style sub-path chunking would help large paths that change slightly, but conflicts with ease of synchronization; skipped.115Encrypted journal on the host:: journald has no built-in journal encryption (FSS sealing is tamper-evidence only), which is why logs are exported and encrypted on the way to the stick instead.116GC of old generations:: not a daemon problem.117Hiding cross-host linkage:: the naming secret is shared project-wide for dedup, so an observer of a shared stick can tell which files two hosts have in common (not what they are). Per-host secrets would hide this at the cost of duplicating shared paths; not a concern for the current clients.