Single Service VPN in NixOS

I recently wanted to set up a service on my server that'd needed to use a VPN for its out of LAN network communication for privacy reason. While also making sure nothing else accidentally used that VPN. The VPN provider I'm using provided a WireGuard configuration for me to use as my starting point. I'm already using WireGuard and know how to set that up on NixOS, but unlike that situation this WireGuard tunnel accepts traffic to any IP, not just a private subnet. This left me with to figure out have to have a single program use the VPN while not having the rest of my system route traffic through a WireGuard interface that accepts any traffic.

So my requirements were such:

1. Have a single service route traffic through the VPN.

2. Do not let any other program on the host use the VPN.

3. Allow the host the access the service's web interface.

Note, this setup may not work if your VPN's address is a domain name instead of an IP address.

networking.wireguard.interfaces.vpn = {
  ips = [ "...." ];
  privateKeyFile = "/path/to/private/key";
  peers = [
    {
      publicKey = "....";
      presharedKeyFile = "/path/to/preshared/key";
      allowedIPs = [
        "0.0.0.0/0" # Accepts and IPv4 or IPv6 address
        "::/0"
      ];
      endpoint = "...";
      persistentKeepalive = 15;
    }
  ];
};

I started with the following basic NixOS WireGuard configuration that mirrors the config file given to me by my VPN provider. The problem with this is config as is, is that it will route all traffic through the VPN interface since it accepts. The assuming that all allowed traffic should be routed through the VPN is correct for VPNs that access private subnets or one used globally on the host system, but no in our case.

My solution to this problem is to use network namespaces[1] to separate the service and the VPN from the rest of the system. By creating different namespaces in Linux, it is possible to give some process access to certain system resources while hiding those resources from others. This is one of the fundamental technologies that Linux containers are built on. Linux provides many different types of namespaces for different types of resources. In our case we just care about network namespaces, which hold network interfaces.

By creating the WireGuard interface in a separate network namespace, it will be hidden from the main system, fulfilling requirement 2[1]!

# Service to set up network namespace
systemd.services."netns@" = {
  description = "%I network namespace";
  before = [ "network.target" ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
    ExecStart = ''
      ${pkgs.iproute2}/bin/ip netns add %I
      ${pkgs.iproute2}/bin/ip -n %I link set lo up # This brings loopback
    '';
    ExecStop = "${pkgs.iproute2}/bin/ip netns del %I";
  };
};

First we define a systemd service that creates a network namespace. Our WireGuard service can then depend on this to bring up and tear down the network namespace along with it. I'm also bringing up the network namespace's loopback interface, which may not be strictly necessary but can be useful. Then we configure WireGuard to use the network namespace by adding two things.

networking.wireguard.interfaces.vpn = {
  ips = [ "...." ];
  privateKeyFile = "/path/to/private/key";
  interfaceNamespace = "vpn";
  peers = [
    {
      publicKey = "....";
      presharedKeyFile = "/path/to/preshared/key";
      allowedIPs = [
        "0.0.0.0/0" # Accepts and IPv4 or IPv6 address
        "::/0"
      ];
      endpoint = "...";
      persistentKeepalive = 15;
    }
  ];
};
systemd.services."wireguard-vpn" = {
  bindsTo = [ "netns@vpn.service" ];
  after = [ "netns@vpn.service" ];
};

Line 4[1] tells systemd to create the WireGuard interface in a network namespace call `vpn` and lines 18-21[2] create the network namespace `vpn` before WireGuard starts.

Technically: lines 18-21 make the wireguard-vpn service (the WireGuard interface creation service added by NixOS) depend on `netne@vpn.service` which creates a network namespace called `vpn`. that WireGuard is expecting. Yay systemd!

systemd.services."" =
  lib.mkIf config.services."" .enable {
    bindsTo = [ "netns@vpn.service" ];
    after = [ "netns@vpn.service" ];
    serviceConfig.NetworkNamespacePath = "/var/run/netns/vpn";
};

Now that we have our network namespace, we just have to modify our service's systemd unit to have it run in our network namespace. Fortunately systemd makes this easy; all we have to do is add the key `NetworkNamespacePath` pointing to the file path of out network namespace. We also make sure the network namespace exists on start up, same as we did for WireGuard.

You may also need to add a namespace specific resolv.conf, like below, if your normal nameserver is not accessible by the continuer.

environment.etc."netns/vpn/resolv.conf".text = ''
  nameserver 8.8.8.8
  nameserver 8.8.4.4
'';

With this done fulfilling requirement 1[1]!. Our service can only use the VPN, as that is the only network interface it has access to (other than loopback) and nothing else can access the VPN. However, we have created a problem we now have to solve. We cannot access the web portal for our service from our browser on our host. Our service is listening on the interfaces in the `vpn` network namespace, none of which we have access to outside of that namespace. To remedy this we'll have to create a way to talk across our namespaces.

networking.wireguard.interfaces.vpn = {
  ips = [ "...." ];
  privateKeyFile = "/path/to/private/key";
  interfaceNamespace = "vpn";
  postSetup = ''
    # Set up virtual Ethernet adapter between normal and VPN namespace to allow nginx to reach slskd
    ${pkgs.iproute2}/bin/ip link add vpn-veth0 type veth peer name vpn-veth1
    ${pkgs.iproute2}/bin/ip link set vpn-veth1 netns vpn
    ${pkgs.iproute2}/bin/ip addr add 192.168.99.1/24 dev vpn-veth0
    ${pkgs.iproute2}/bin/ip link set vpn-veth0 up
    ${pkgs.iproute2}/bin/ip netns exec vpn ip addr add 192.168.99.2/24 dev vpn-veth1
    ${pkgs.iproute2}/bin/ip netns exec vpn ip link set vpn-veth1 up
  '';
  preShutdown = ''
    # Tear down virtual Ethernet adapter
    ${pkgs.iproute2}/bin/ip link del dev vpn-veth0
  '';
  peers = [
    {
  	  publicKey = "....";
  	  presharedKeyFile = "/path/to/preshared/key";
  	  allowedIPs = [
  	    "0.0.0.0/0" # Accepts and IPv4 or IPv6 address
  	    "::/0"
  	  ];
  	  endpoint = "...";
      persistentKeepalive = 15;
    }
  ];
};

We modify the WireGuard config to create a virtual ethernet link between our default network namespace and the `vpn` namespace; assigning IP address to both ends. With this we can now access our service's web interface at `http://192.168.99.2:<port>/`. We have now fulfilled requirement 3[1]! That's all of them!

We now have a working config, but I'll break down those setup command more before moving on.

# Create a virtual ethernet interface pair named vpn-veth0 and vpn-veth1
ip link add vpn-veth0 type veth peer name vpn-veth1
# Move vpn-veth1 into the vpn network namespace
ip link set vpn-veth1 netns vpn
# Assign vpn-veth0 the IP address 192.168.99.1 (chose arbitrary)
ip addr add 192.168.99.1/24 dev vpn-veth0
# Bring vpn-veth0 up
ip link set vpn-veth0 up
# Assign vpn-veth1 the IP address 192.168.99.2 (chose arbitrary)
ip netns exec vpn ip addr add 192.168.99.2/24 dev vpn-veth1
# Bring vpn-veth1 up
ip netns exec vpn ip link set vpn-veth1 up

Here is the complete config with an Nginx reverse proxy added. You may wish to use a different reverse proxy, but if want a proper domain instead of going to `http://192.168.99.2:<port>/` you may want one.

services."".enable = true; # Enable and configure whatever service you are using

# Have service use the VPN namespace
systemd.services."" =
  lib.mkIf config.services."".enable {
    bindsTo = [ "netns@vpn.service" ];
    after = [ "netns@vpn.service" ];
    serviceConfig.NetworkNamespacePath = "/var/run/netns/vpn";
};

# Configure the VPN
networking.wireguard.interfaces.vpn = {
  ips = [ "...." ];
  privateKeyFile = "/path/to/private/key";
  interfaceNamespace = "vpn";
  postSetup = ''
    # Set up virtual Ethernet adapter between normal and VPN namespace to allow nginx to reach slskd
    ${pkgs.iproute2}/bin/ip link add vpn-veth0 type veth peer name vpn-veth1
    ${pkgs.iproute2}/bin/ip link set vpn-veth1 netns vpn
    ${pkgs.iproute2}/bin/ip addr add 192.168.99.1/24 dev vpn-veth0
    ${pkgs.iproute2}/bin/ip link set vpn-veth0 up
    ${pkgs.iproute2}/bin/ip netns exec vpn ip addr add 192.168.99.2/24 dev vpn-veth1
    ${pkgs.iproute2}/bin/ip netns exec vpn ip link set vpn-veth1 up
  '';
  preShutdown = ''
    # Tear down virtual Ethernet adapter
    ${pkgs.iproute2}/bin/ip link del dev vpn-veth0
  '';
  peers = [
    {
  	  publicKey = "....";
  	  presharedKeyFile = "/path/to/preshared/key";
  	  allowedIPs = [
  	    "0.0.0.0/0" # Accepts and IPv4 or IPv6 address
  	    "::/0"
  	  ];
  	  endpoint = "...";
      persistentKeepalive = 15;
    }
  ];
};

# Define a service for creating network namespaces
systemd.services."netns@" = {
  description = "%I network namespace";
  before = [ "network.target" ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
    ExecStart = ''
      ${pkgs.iproute2}/bin/ip netns add %I
      ${pkgs.iproute2}/bin/ip -n %I link set lo up # This brings loopback
    '';
    ExecStop = "${pkgs.iproute2}/bin/ip netns del %I";
  };
};

# Nginx reverse proxy
services.nginx.virtualHosts."" = {
  # Example params to enable https, not required
  forceSSL = true;
  enableACME = true;

  locations."/" = {
    # Point nginx on the vpn network namespace's virtual Ethernet IP address
    proxyPass = "http://192.168.99.2:${toString config.services."".settings.port}";
	# Add any other reverse proxy header, etc. here
	# Consult service docs on what is needs
  };
};