Viktor Vilhelm Sonesten

Reviving my NAS with flash drives and copytoram

Published on

NixOS 25.11 recently released and I upgraded by systems. Unfortunately my headless NAS stopped responding after the reboot and my infrastructure now depended on it, so it was time to put on the debugging hat. It turned out that the boot SSDs (yes, both) kicked the bucket and threw the system into a boot loop. I had no spare SSDs available but did have flash drives, and I have been working with disk-less systems a lot this year. So I figured I might as well try the approach out on my own systems. Here I cover what I did, what I did wrong, and what's next.

First up was keys. For SSH and agenix I didn't want to rekey on every boot. Aside from that, Tailscale has a tailscaled.state it wants to keep around. After some iterations I ended up with a /persist file system:

  $ tree /persist
/persist
├── keys
│   └── ssh
│       ├── ssh_host_ed25519_key
│       ├── ssh_host_ed25519_key.pub
│       ├── ssh_host_rsa_key
│       └── ssh_host_rsa_key.pub
├── lost+found
└── tailscaled.state

4 directories, 5 files

I arrived at this magnificant solution after a first try at creating a sidecar keys dataset on the zpool the NAS manages… that otherwise contains an encrypted dataset… which requires a key decrypted by the host SSH keys… which put me in a catch 22 were I suddenly depended on initial zpool imports to fail (but only on the first try) and then hacking around with agenix to-

Simple usually means more of the same, so through the magic of buying two of them, I now have a small subset of state that the read-only system can utilize; all user data continues to live in the zpool.

With host keys now pinned again I agenix --rekey and modify my system configuration:

  • fileSystems is reduced down to just mounting a /persist, services.openssh.hostKeys is pointed there, and Tailscale is told where to keep track of state;
  • swap devices are removed;
  • the grub bootloader is disabled;
  • the copytoram kernel parameter is added; and
  • I wrap the configuration in a iso-image.nix import and config.system.build.isoImage build.

The diff itself can be of use (and is surprisingly small), so its copied (almost) verbatim down below.

copytoram is not strictly required but I prefer my flash drives not running hot in an always-on system — and otherwise above-room-temp server room — and it allows me to pull the drive, write the new system image, reinsert it, and issue a reboot without issues.

And that's it! My NAS is back up and running again: "installing" the system is as easy as nix build && dd. Big thanks to NixOS and all its contributors for making this so easy!

Next up

This was all just to get my NAS back on its feet again. There are some things I'd like to do in order to further stabilize it:

  • buy a new SSD that will act as /persist — so that I can have it inside the chassi; and
  • provide swap (most likely via the same SSD lest openzfs can figure it out).

But the biggest item is to boot into iPXE instead and load kernel/initramfs over network. I plan to replace my openwrt router with something more (NixOS-)capable, and I'd like to implement a CI/CD solution that allows a simple reboot to fetch the latest image for a given system. That way I can bootstrap my infrastructure from a single system and minimize dynamic state overall.

You know, for fun.

Patch of changes

 nix/modules/systems/default.nix               | 17 +++++++++++
 nix/modules/systems/dulcia/boot-copytoram.nix | 30 +++++++++++++++++++
 nix/modules/systems/dulcia/default.nix        |  1 +
 .../systems/dulcia/hardware-configuration.nix | 16 ----------
 nix/modules/systems/dulcia/nas.nix            |  4 ---
 5 files changed, 48 insertions(+), 20 deletions(-)
 create mode 100644 nix/modules/systems/dulcia/boot-copytoram.nix

diff --git a/nix/modules/systems/default.nix b/nix/modules/systems/default.nix
--- a/nix/modules/systems/default.nix
+++ b/nix/modules/systems/default.nix
@@ -27,4 +27,21 @@ in {
+
+  perSystem = { config, ... }: {
+    packages.dulcia-boot = (nixosSystem {
+      system = "x86_64-linux";
+      modules = [
+        nixosModules.dulcia
+        opkgs
+
+        # For the isoImage attribute
+        ({ modulesPath, ... }: {
+          imports = [ "${modulesPath}/installer/cd-dvd/iso-image.nix" ];
+          isoImage.makeBiosBootable = true;
+          isoImage.makeUsbBootable = true;
+        })
+      ];
+    }).config.system.build.isoImage;
+  };
 }
diff --git a/nix/modules/systems/dulcia/boot-copytoram.nix b/nix/modules/systems/dulcia/boot-copytoram.nix
new file mode 100644
--- /dev/null
+++ b/nix/modules/systems/dulcia/boot-copytoram.nix
@@ -0,0 +1,30 @@
+# Settings for dulcia so that she can boot from USB and copy the
+# operating system into RAM.
+{ config, lib, ... }: {
+  boot.kernelParams = [ "copytoram" ];
+
+  fileSystems."/persist" = {
+    device = "/dev/disk/by-label/persist";
+
+    # Agenix decrypts in stage-2-init; ensure /persist is mounted by
+    # then.
+    neededForBoot = true;
+  };
+
+  # Make tailscale store state (keys) in /persist
+  services.tailscale.extraDaemonFlags = [ "--state=/persist/tailscaled.state" ];
+
+  # Copy of the default settings, but the path has been changed to
+  # /persist.
+  services.openssh.hostKeys = [
+    {
+      bits = 4096;
+      path = "/persist/keys/ssh/ssh_host_rsa_key";
+      type = "rsa";
+    }
+    {
+      path = "/persist/keys/ssh/ssh_host_ed25519_key";
+      type = "ed25519";
+    }
+  ];
+}
diff --git a/nix/modules/systems/dulcia/default.nix b/nix/modules/systems/dulcia/default.nix
--- a/nix/modules/systems/dulcia/default.nix
+++ b/nix/modules/systems/dulcia/default.nix
@@ -12,6 +12,7 @@
       ./forgejo.nix
       ./miniflux.nix
+      ./boot-copytoram.nix
     ];
diff --git a/nix/modules/systems/dulcia/hardware-configuration.nix b/nix/modules/systems/dulcia/hardware-configuration.nix
--- a/nix/modules/systems/dulcia/hardware-configuration.nix
+++ b/nix/modules/systems/dulcia/hardware-configuration.nix
@@ -15,22 +15,6 @@

   boot.zfs.extraPools = [ "stable" ];

-  fileSystems."/" =
-    { device = "root";
-      fsType = "zfs";
-    };
-
-  fileSystems."/boot" =
-    { device = "/dev/disk/by-uuid/BBBC-3DCA";
-      fsType = "vfat";
-      options = [ "fmask=0077" "dmask=0077" ];
-    };
-
-  swapDevices =
-    [ { device = "/dev/disk/by-uuid/7ff36e83-6019-42dc-aec5-82b35c10cc70"; }
-      { device = "/dev/disk/by-uuid/8f930bf3-08ee-4686-8e81-c200ef97937f"; }
-    ];
-
diff --git a/nix/modules/systems/dulcia/nas.nix b/nix/modules/systems/dulcia/nas.nix
--- a/nix/modules/systems/dulcia/nas.nix
+++ b/nix/modules/systems/dulcia/nas.nix
@@ -1,8 +1,4 @@
 { config, modulesPath, ... }: {
-  boot.loader.grub = {
-    enable = true;
-    devices = [ "/dev/sda" "/dev/sdb" ];
-  };