Mounting a nix-store via NFS

Lately at $work we wanted to netboot some (NixOS) Computers. Having a full nix-store for each one on some Server seemed wasteful though, so I searched for another solution.

NixOS does not need a full rootfs to boot, it just needs /nix (and the pseudo file systems like /dev, /proc,...). All the other directories get generated inside the activation script.

The path seemed clear:

1. Netboot the kernel/initramfs from HTTP/TFTP
2. Mount the nix-store via NFS inside the initramfs
3. Profit

Multiple instances of nix writing onto the same store might mean problems. So we'll just mount the nix store read only.

| But we might want to write to it

So we'll overlay a rw directory over the read only store.

I just naively tried

{ ... }: {
  boot.initrd.supportedFilesystems = [ "nfs" "nfsv4" "overlay" ];   # load needed kernel modules
  boot.initrd.availableKernelModules = [ "nfs" "nfsv4" "overlay" ]; # load them again, because of cause it didn't work
  fileSystems."/" = { device = "tmpfs"; fsType = "tmpfs"; options = [ "size=2G" ]; };
  fileSystems."/nix/.rw-store" = { fsType = "tmpfs"; options = [ "mode=0755" "size=8G" ]; neededForBoot = true; };
  fileSystems."/nix/.ro-store" =
    {
      fsType = "nfs4";
      device = "192.168.1.1:nixstore";
      options = [ "ro" ];
      neededForBoot = true;
    };
  fileSystems."/nix/store" =
    {
      fsType = "overlay";
      device = "overlay";
      options = [
        "lowerdir=/nix/.ro-store"
        "upperdir=/nix/.rw-store/store"
        "workdir=/nix/.rw-store/work"
      ];
      depends = [
        "/nix/.ro-store"
        "/nix/.rw-store/store"
        "/nix/.rw-store/work"
      ];
    };
  boot.initrd.network.enable = true;
  networking.useDHCP = true;
}

but it just didn't boot. The initramfs ran through, but then it just hung.

NixOS initramfs networking yeets the network configuration after booting successfully.

You have to set boot.initrd.network.flushBeforeStage2 to false to disable that.

And now it just works™

Just for completeness, a full configuration is:

{ ... }: {
  boot.initrd.supportedFilesystems = [ "nfs" "nfsv4" "overlay" ];   # load needed kernel modules
  boot.initrd.availableKernelModules = [ "nfs" "nfsv4" "overlay" ]; # load them again, because of cause it didn't work
  fileSystems."/" = { device = "tmpfs"; fsType = "tmpfs"; options = [ "size=2G" ]; };
  fileSystems."/nix/.rw-store" = { fsType = "tmpfs"; options = [ "mode=0755" "size=8G" ]; neededForBoot = true; };
  fileSystems."/nix/.ro-store" =
    {
      fsType = "nfs4";
      device = "192.168.1.1:nixstore";
      options = [ "ro" ];
      neededForBoot = true;
    };
  fileSystems."/nix/store" =
    {
      fsType = "overlay";
      device = "overlay";
      options = [
        "lowerdir=/nix/.ro-store"
        "upperdir=/nix/.rw-store/store"
        "workdir=/nix/.rw-store/work"
      ];
      depends = [
        "/nix/.ro-store"
        "/nix/.rw-store/store"
        "/nix/.rw-store/work"
      ];
    };
  boot.initrd.network.enable = true;
  boot.initrd.network.flushBeforeStage2 = false; # otherwise nfs dosen't work
  networking.useDHCP = true;
}

The netboot stuff we built is open source and available here, but not ready for production yet.