sops secrets in the berry patch

NixOS deploys the configuration, sops-nix deploys the secrets, and a YubiKey guards the key that gets three Raspberry Pis started.

I have three Raspberry Pis called blackberry, blueberry, and cranberry. They are headless, scattered around the house, and nowhere near Ethernet. Each needs the Wi-Fi password before it can do anything useful.

They run NixOS. Their configuration and encrypted secrets live together in a public git repository. Nix builds each operating-system image; sops-nix decrypts that machine’s secrets when it boots. This is pleasantly uneventful once the machine has an identity. A blank SD card does not.

The berry bootstrap

Each Pi uses its SSH host key as its sops age identity. Blackberry’s Wi-Fi password, for example, is encrypted to the age recipient derived from blackberry’s host key. At boot, sops-nix uses the private host key to decrypt the password and gives it to the NixOS wireless configuration.

The dependency graph is unfortunately circular:

  • The Pi needs the Wi-Fi password to join the network.
  • The password is encrypted to the Pi’s SSH host key, which is how sops-nix lets a machine decrypt its own secrets at boot.
  • The host key is generated on first boot.
  • First boot requires the network.

Ethernet would break the cycle, but running cable to three berries is an unsatisfying answer to a secrets problem. The host key has to arrive before the first boot.

Nix builds the image, but never sees the key

The obvious move is to give the host key to Nix and bake it into the image. Nix is already assembling the filesystem. It would be very good at this and should not be allowed anywhere near it.

Anything a Nix build produces lands in /nix/store, which is world-readable by design. Store paths can also reach binary caches and persist long after the original build. A private key cannot enter the build.

Ciphertext can enter the image because ciphertext in the store is inert. The plaintext host key arrives afterward through a local provisioning step outside Nix:

# reproducible, cacheable, secret-free
nix build .#packages.aarch64-linux.blackberry-pi-image

# loop-mounts it and places the one key Nix must never hold
bin/pi-image-provision blackberry-pi /var/tmp/blackberry-pi.raw

The complete trip from git to a booting berry is:

  1. Nix builds a reproducible, secret-free NixOS image.
  2. The provisioning command decrypts that Pi’s host key and writes it into /etc/ssh on the image.
  3. The Pi boots with its permanent SSH identity already in place.
  4. sops-nix uses that identity to decrypt the Wi-Fi password and the other secrets addressed to that machine.
  5. NixOS starts the services that consume them.

Blackberry, blueberry, and cranberry each have a different host key and age recipient. Their NixOS configurations can share most of a module tree, but their provisioned images must differ in /etc/ssh. Blackberry has no business reading cranberry’s mail.

The key that provisions the keys

The provisioning step still has to decrypt the sops document containing each host key. That creates one more identity to protect.

If that identity is a private key in ~/.config, copying one laptop file is enough to provision every Pi. age-plugin-yubikey instead keeps the identity key on a YubiKey. sops finds the plugin stub on disk, the plugin asks the YubiKey to perform the private-key operation, and the key remains on the device.

The repository and the stub can travel freely. Provisioning still requires the YubiKey, its PIN, and a touch. The Nix build never sees a plaintext key, and the laptop never stores the identity that can release one.

When the YubiKey looks guilty

My first provisioning attempt failed with:

Recovery failed because no master key was able to decrypt the file.

That message does not distinguish a missing identity from a malformed one, or a YubiKey that was never consulted from one that refused the operation. sops knows only that no identity succeeded.

So I guessed. I decided the problem was a missing SOPS_AGE_KEY_FILE and wrote that into a commit message and a source comment before testing the claim. sops reads $XDG_CONFIG_HOME/sops/age/keys.txt on its own; the variable changes nothing. A ten-line fixture that encrypted to a throwaway key at the default path and decrypted with a bare environment would have disproved the claim in under a minute. I reverted the commit once I finally ran one.

I caused the next failure while editing the identity file. I captured a plugin’s output with 2>&1, which merged stderr into stdout and inserted a diagnostic line into the file. That produced unknown identity type, again nested inside the same outer error.

The decrypt then succeeded, but a jq invocation later in the pipeline failed with Invalid numeric literal at line 1, column 11. Column eleven is the width of -----BEGIN . sops’s --extract returned a raw PEM string to a pipeline expecting JSON.

None of these failures came from the YubiKey. It was merely the most interesting suspect, while the culprits were an unnecessary environment variable, a stray stderr line, and a PEM mistaken for JSON.

Hardware adds weather

Files fail in ways software understands. Hardware adds weather: the key can be absent, waiting for a touch, or locked behind a PIN, while sops reports only that no identity succeeded. The surrounding software must distinguish those states from a missing file, a malformed identity, or invalid output.

Blackberry, blueberry, and cranberry are still on the bench. Before I flash them, the provisioning wrapper needs to distinguish an unreadable identity file from a missing YubiKey, a pending touch, a rejected PIN, and invalid decrypted output.

That requirement is not peculiar to Raspberry Pis. Any NixOS deployment using sops with a hardware identity crosses the same boundary. The berry patch merely made it impossible to ignore: an operator who cannot diagnose a security control will eventually route around it.