The bottom turtle is a hardware key

Encrypting secrets into git is the easy half — the hard half is what opens the first door, and that chain has to bottom out in something you can't copy.

Committing encrypted secrets to a public repository is a solved problem. sops encrypts the values and leaves the keys legible, so a diff still reads like a diff. That part is boring, and boring is the point.

The interesting question is the one underneath: what decrypts the secrets? Whatever answers that needs a key of its own, and so does whatever holds that key. The chain has to bottom out somewhere that isn’t another file, or you have relocated the problem and given it a nicer filename.

I have three Raspberry Pis. They are called blackberry, blueberry, and cranberry, because a naming convention that makes you smile is a naming convention you will still be using in a year. They are headless and they have no Ethernet run, so each one needs the house Wi-Fi password before it can do anything useful, including tell you it failed.

The bootstrap paradox

  • The Pi needs the Wi-Fi password to join the network.
  • The password is encrypted to the Pi’s SSH host key, which is how sops-nix lets a machine decrypt its own secrets at boot.
  • The host key is generated on first boot.
  • First boot requires the network.

Plug in Ethernet and this evaporates, but the whole point was not to. From inside the loop nothing dissolves it: something outside has to reach in and place the first key. The only decision is what.

Not the build, definitely not the build

The obvious move is to bake the host key into the image. Nix builds the image; Nix can write a file; write the key.

Do not do this. Anything a Nix build produces lands in /nix/store, which is world-readable by design and addressed by a stable, public hash. A private key in a store path is a private key with a predictable name, copied to every binary cache you ever pushed to, surviving long after you have forgotten it was there. The store is not a vault. It is a warehouse with the doors open, and that is a feature everywhere except here.

So the encrypted document goes in at build time — ciphertext in the store is inert, which is the entire premise — and the one plaintext secret gets injected afterward, outside Nix, by a step that is local, impure, and deliberately unrepeatable by anyone but me:

# reproducible, cacheable, secret-free
nix build .#packages.aarch64-linux.blackberry-pi-image

# loop-mounts it and places the one key Nix must never hold
bin/pi-image-provision blackberry-pi /var/tmp/blackberry-pi.raw

This is also why the three Pis cannot share one image. Each has its own host key and its own age recipient, so the images differ only in /etc/ssh — but they must differ, or blackberry could read cranberry’s mail.

Which brings us to the turtle

The provisioning step decrypts a document. What opens that?

If the answer is a private key sitting in ~/.config, the security of three machines reduces to the security of one laptop file, and a stolen laptop is a stolen fleet. The alternative sounds like work, which is generally enough to keep it from getting done.

The alternative is a hardware key, and the property that matters is narrow and absolute: the private key cannot leave the device. Not “should not.” Cannot. You do not get a copy. You get an oracle — you hand it ciphertext, it hands back plaintext, and the key stays where it was born. A plugin like age-plugin-yubikey puts a stub in your identity file, a pointer of no value to a thief, and the key itself never touches the disk it protects.

Which means the identity file, the repo, and the images are all safely public in the only sense that counts: possessing them gets you nothing. The chain of keys bottoms out in an object that must be physically present and physically touched. The turtle at the bottom is a piece of metal in your pocket.

The part where the turtle appears to be dead

Here is the tax, and it is the reason I wrote this down.

Everything above is the part that worked. Then I ran the command and got:

Recovery failed because no master key was able to decrypt the file.

That message is the entire diagnostic. It does not separate a key that was never asked from a key that refused, or a malformed identity file from an absent one. From sops’s altitude “no identity matched” is the whole truth, and the cause lives several layers underneath it.

So I guessed. I decided the problem was a missing SOPS_AGE_KEY_FILE and wrote that into a commit message and a source comment before testing the claim. sops reads $XDG_CONFIG_HOME/sops/age/keys.txt on its own; the variable changes nothing. A ten-line fixture — encrypt to a throwaway key at the default path, decrypt with a bare environment — would have caught it in under a minute, and I reverted the commit once I finally ran one.

The next failure I caused outright. Editing the identity file, I captured a plugin’s output with 2>&1 — merging stderr into stdout, interleaving a diagnostic line into the middle of the file, and producing unknown identity type nested inside that same outer message.

Then the decrypt succeeded and a jq further down the pipeline exploded on Invalid numeric literal at line 1, column 11 — column eleven being exactly the width of -----BEGIN . sops’s --extract returns a raw string; the pipeline had asked for JSON and been handed a PEM.

None of it was the hardware key. A plausible mechanism that explains the symptom is not the mechanism, and hardware is where that distinction goes soft — the physical explanation is always available, and it always feels sufficient.

Budget for the physical layer

Hardware-backed secrets are not harder cryptography. The cryptography is identical. What changes is that failure acquires a physical dimension — a key in the wrong port, an unpressed touch, a PIN cached in a session you closed — and none of your software knows the physical world exists.

The berries are still on the bench, waiting to be flashed, and I am no longer in a hurry about it. The repo’s own sops-edit wrapper turned out to carry a comment about one of these traps: someone had been here before and left a note. Write the diagnostic that names the actual missing piece. That is part of the security design, not a convenience on top of it — a control nobody can operate is a control that will be routed around by a tired person at midnight.