capsule.adrianhesketh.com

home

Setting up a NixOS remote builder for the M1 Mac

If you want to build a Virtual Machine disk image, or Docker container using Nix on an M1 Mac, you'll need to use a remote builder that's running Linux.

Virtual machines

We'll set up two Linux machines, one for ARM64, and another for x86_64, using UTM.

ISO

Download the Minimal NixOS ISO for both ARM64 and x86_64 from [0].

[0]

Usage in UTM

Create a new virtual machine in UTM.

Next, choose "Other" as the operating system, so that you can select the downloaded ISO files.

Networking

Customise the networking to use "Bridged" mode, so that the VMs can be accessed from the host machine [1].

Note that the interface is explicitly set to `en0` on the Mac, which is the WiFi interface. If you're using Ethernet, you may need to change this to `en1`.

[1]

Installation

When NixOS starts, you'll be dropped into a shell, but the VM disk won't be mounted.

You can see the VM disk by running `lsblk`.

For some reason, in UTM, the disk is `/dev/vda` for aarch64, while it's `/dev/sda` for the emulated x86_64 machine.

NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0   7:0    0 899.1M  1 loop /nix/.ro-store
sr0    11:0    1 984.9M  0 rom  /iso
vda   254:0    0  128G  0 disk

The commands all need to run as root, so run `sudo -i` to become root.

Use `cfdisk` to partition the disk. Use `/dev/vda` for the ARM64 machine, and `/dev/sda` for the x86_64 machine.

cfdisk /dev/vda

Create EFI partition

Create swap partition

Create root partition

Results

Select "Write" to write the changes, then quit.

List of partitions

Create filesystems

Create the filesystems for the partitions. Remember to use `/dev/vda` for the ARM64 machine, and `/dev/sda` for the x86_64 machine.

mkfs.fat -F 32 -n boot /dev/vda1
mkswap -L swap /dev/vda2
mkfs.ext4 -L nixos /dev/vda3

Mount the filesystems

mount /dev/vda3 /mnt
mkdir -p /mnt/boot
mount /dev/vda1 /mnt/boot

Enable the swap.

swapon /dev/vda2

Create a root SSH user

An SSH key and username are set up in my NixOS configuration in Github, but note that I have two SSH keys, one for my normal user, and one for root, which actually carries out Nix build operations on my Mac.

To generate the SSH key for the root user on your Mac, use `ssh-keygen` as root, then get the public key to add to your NixOS configuration.

sudo ssh-keygen

The public key is in `/var/root/.ssh/id_ed25519.pub`, `cat` that file to get the public key.

Download and build configuration

The configuration at https://github.com/a-h/nixos includes a flake that can be used to build a NixOS system for `aarch64` and `x86_64`.

Although the machines have different mount points for disks, the NixOS configuration looks up the disk by label, so the same configuration can be used for both machines.

fileSystems."/" = {
  device = "/dev/disk/by-label/nixos";
  fsType = "ext4";
};
fileSystems."/boot" = {
  device = "/dev/disk/by-label/boot";
  fsType = "vfat";
};
swapDevices = [
  {
    device = "/dev/disk/by-label/swap";
  }
];

To build the config for `aarch64`, run the following commands, replace `builder-aarch64` with `builder-x86_64` for the x86_64 machine.

curl -LO https://github.com/a-h/nixos/archive/refs/heads/main.zip
unzip ./main.zip
cd nixos-main && nixos-install --root /mnt --flake .#builder-aarch64

Shutdown

shutdown now

Remove the ISO, and restart

Remove the boot ISO from the VM, and restart the VM.

SSH into the machines

To find the IP address of the virtual machines, run `ip a` on them.

I set up a DNS entry in `/etc/hosts` on the Mac, so that I can easily SSH into the machines.

192.168.0.62 builder-aarch64
192.168.0.63 builder-x86_64

Then I can SSH into the machines using a hostname.

ssh adrian@builder-aarch64

Since the username on my Mac and the VMs is the same, I don't need to specify the username.

ssh builder-aarch64

Using the remote builder from MacOS

The documentation at [2] explains how to set up a remote builder.

[2]
nix store info --store ssh://builder-aarch64

To ensure that Linux builds are done on the corresponding VM for the architecture, set up the builders in the `nix.conf` file. On my machine it's in `~/.config/nix/nix.conf`.

builders = ssh://adrian@builder-x86_64 x86_64-linux ; ssh://adrian@builder-aarch64 aarch64-darwin

Remember that the build will be done by your Nix daemon on your local machine.

If you see:

1 146 /Users/adrian % nix build --impure \
  --expr '(with import  { system = "x86_64-linux"; }; runCommand "foo" {} "uname > $out")'
cannot build on 'ssh://builder-x86_64': error: failed to start SSH connection to 'builder-x86_64': Host key verification failed.
Failed to find a machine for remote build!
derivation: y1i89c0934r6v492a12px1p5h44h7481-foo.drv
required (system, features): (x86_64-linux, [])
2 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux], 1, [], [])
([aarch64-darwin], 1, [], [])
error: a 'x86_64-linux' with features {} is required to build '/nix/store/y1i89c0934r6v492a12px1p5h44h7481-foo.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Note, the `Host key verification failed.` message.

To get the host keys into the root user, copy the values from your local user's known hosts file (`~/.ssh/known_hosts`) to the root user's known hosts file (`/var/root/.ssh/known_hosts`), or run `sudo ssh adrian@builder-x86_64` to manually add each one.

Then run the build command again.

You may run into the issue that the `root` user on your machine doesn't have the same SSH keys as _your_ user. Either create an SSH key for the `root` user on your Mac (as described earlier), and update the remote builder to support the use of that key, or copy the SSH keys from your normal user to the Mac `root` user.

sudo cp -r ~/.ssh /root

Testing

To test that the remote builder is working, run the following command.

nix build --impure \
  --expr '(with import  { system = "x86_64-linux"; }; runCommand "foo" {} "uname > $out")'

The `./result` file should contain `Linux`.

Similarly, for the ARM64 machine.

nix build --impure \
  --expr '(with import  { system = "aarch64-linux"; }; runCommand "foo" {} "uname > $out")'

More

Previous

DynamoDB diagrams from text

Home

home