Encrypted NixOS home server with passwordless reboot

These are my notes on refurbishing a laptop with a broken screen hinge to a NixOS home server. A coworker recommended Colmena for managing NixOS on remote machines, so I decided to give it a try. I got confused by the Colmena manual, which expects NixOS to be already set up on the remote host but doesn't clearly show how to move the existing nix (remote) config inside Colmena.

Initial NixOS setup

First, NixOS must be set up on the target machine. I followed the official documentation, including full-disk encryption.

After the initial reboot, a couple of adjustments were needed in /etc/nixos/configuration.nix:

• choosing a networking.hostName
• enabling the ssh server (services.openssh.enable)
• allowing passwordless sudo for my user (security.sudo.extraRules)

I then rebuilt the system `nixos-rebuild switch` and was able to log in from my main computer via ssh using the password. Once connected via ssh:

• add the ssh public key to `users.users.<username>.openssh.authorizedKeys.keys` of the target computer and rebuild
• scp both files from the remote /etc/nixos/ into a dedicated `host-a` folder in the colmena git repo on my main computer
• adjust the colmena flake.nix config to import the configuration.nix file (which imports the hardware-configuration.nix file):

host-a = {
  deployment = {
    targetHost = "192.168.1.14";
    targetPort = 22;
    targetUser = "";
    buildOnTarget = true;
  };
  time.timeZone = "Europe/Berlin";
  imports = [
    ./host-a/configuration.nix
  ];
};

After that, `colmena apply` should be able to connect to the remote and deploy the config.

Bypassing SSH interactive auth

I use a TPM-backed ssh key which asks for a pin on every connection. To workaround the (documented) limitation of Colmena which requires non-interactive login, I started a ssh connection in "master mode" in another terminal. With this command running in the background, I am now able to run `colmena apply`.

ssh -M username@host-a

Passwordless reboot

Since I setup a full-disk encryption, I need to type the password on every boot. However I read recently on Lobster's that it was possible to skip this, when rebooting with kexec.

I first tried rebooting with `systemctl kexec`, but got the error

No kexec kernel loaded and autodetection failed.
Automatic loading works only on systems booted with EFI.

Since my laptop boots with a BIOS, this error was somewhat explainable. I (much later) saw, that it was a nixpkgs/systemd issue, since `systemctl start kexec.target` worked fine.

Anyway, I found a couple of helpful resources online to prepare kexec and after a lot of trial and error, I was finally able to reboot without a password!

The final systemd service looks like this (the prepare-kexec.service is managed by NixOS: it prepares an initrd image if not done already, which we are doing):

  systemd.services."kexec-load" = {
    unitConfig.DefaultDependencies = false;
    before = [ "prepare-kexec.service" ];
    path = with pkgs; [ cpio gzip kexec-tools ];
    script = (builtins.readFile ../scripts/kexec-load.sh);
    postStop = (builtins.readFile ../scripts/kexec-load-cleanup.sh);
    serviceConfig.Type = "oneshot";
    wantedBy = [ "kexec.target" ];
  };

And the kexec-load.sh reads:

#!/bin/sh

set -e
# adapted from https://github.com/flowztul/keyexec/blob/9af064a6aa4d92cc42d062398bc80754a9a6edd8/etc/default/kexec-cryptroot
# Copyright 2017, Lutz Wolf flow@0x0badc0.de
# Licensed under GPLv2 or later.

# Generate temporary initrd.img with LUKS master keys for kexec reboot

umask 0077

CRYPTROOT_TMPDIR="$(mktemp -d --tmpdir=/dev/shm initrd.XXXXXXXXXX)"

# clear the folder when exiting
cleanup() {
    shred -fu "${CRYPTROOT_TMPDIR}/initrd.img" || true
    shred -fu "${CRYPTROOT_TMPDIR}/boot/crypto_keyfile.bin" || true
    rm -rf "${CRYPTROOT_TMPDIR}"
}

trap cleanup INT TERM EXIT

p=$(readlink -f /nix/var/nix/profiles/system)
if ! [[ -d $p ]]; then
  echo "Could not find system profile for prepare-kexec"
  exit 1
fi

# prepare the boot folder
mkdir "${CRYPTROOT_TMPDIR}/boot"
cp /boot/crypto_keyfile.bin "${CRYPTROOT_TMPDIR}/boot/"

# append the boot folder to the initrd
cp "$p/initrd" "${CRYPTROOT_TMPDIR}/initrd.img"
cd "${CRYPTROOT_TMPDIR}"
find boot | cpio -H newc -o | gzip >> "${CRYPTROOT_TMPDIR}/initrd.img"

# load the new initrd (exec is needed)
exec kexec --load "$p/kernel" --initrd="${CRYPTROOT_TMPDIR}/initrd.img" --append="$(cat "$p/kernel-params") init=$p/init"

Running kexec (without exec) would succeed when running directly, but fail when run via systemctl. So the cleanup had to happen in another script (apparently trap EXIT does not work with exec), which I named kexec-load-cleanup.sh:

#!/bin/sh

PREFIX="/dev/shm/initrd."

for dir in ${PREFIX}*/; do
    shred -fu "${dir}/initrd.img" || true
    shred -fu "${dir}/boot/crypto_keyfile.bin" || true
    rm -rf "${dir}"
done

This cleanup is likely not needed since the files were created in RAM and are discarded on reboot. Better safe (with shred) than sorry since we are playing with LUKS keys.

Colmena reboot

Colmena does not expose a configuration to choose how to reboot, but directly executes `reboot`. So I created a `reboot` package in `environment.systemPackages`

(pkgs.writeShellScriptBin "reboot" "systemctl start kexec.target")

Et voilà, I can now run `colmena apply --reboot` and get a rebooted server, without user interaction!

📆 2025-01-19