Using flakes for NixOS configurations

— Stéphan Kochen, 2023-02-28

All of my personal projects run on a small Hetzner cloud server with NixOS. I recently converted its configuration to use flakes, for a couple of reasons:

  • Many of the projects I deploy are already flakes themselves. I was using flake-compat to import them in the old setup, but had to manually update hashes on every update.

  • I also have some private projects in Git that I only ever sync over SSH, and aren't on any hosted Git service. Deploying these was a giant hack, basically by creating a prebuilt tarball outside Nix. I really wanted these to be simple flake inputs as well.

In this post, I'll talk about basic steps, auto-upgrades, and the challenges deploying those private projects.

A basic flake

Given an existing configuration.nix for a machine, wrapping it in a flake is quite simple. Simply place a flake.nix next to it containing something like:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
  };
  outputs = { nixpkgs, ... }: {
    nixosConfigurations.gerbil = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ ./configuration.nix ];
    };
  };
}

Now, nixos-rebuild will simply work as before, IF you stick to two conventions:

  • The flake is located at: /etc/nixos/flake.nix
    (But it may also be a symlink pointing to the actual location.)

  • The hostname matches the nixosConfigurations attribute name. (gerbil in this example.)

Otherwise, you need to use the --flake flag:

nixos-rebuild switch --flake '/path/to/flake/directory#hostname`

Automatic upgrades

Normally you'd do something like the following in your configuration to receive automatic upgrades of the base NixOS system and packages:

system.autoUpgrade.enable = true;

But this auto-upgrade system relies on channels, and channels are impure, and impurities are banned in the land of flakes. But all it takes to get auto-upgrades back is a couple extra flags:

system.autoUpgrade = {
  enable = true;
  flake = "/etc/nixos#gerbil";
  flags = [ "--update-input" "nixpkgs" ];
};

Note that I set flake explicitly. Even if you stick to the default location, that's necessary, because the auto-upgrade module isn't smart enough to detect yes or no flakes.

The key part to actually getting updates is adding the --update-input flag here. You can pass this multiple times for any other inputs you want to update automatically, or you can use --recreate-lock-file to update everything.

Remote builds

I like to manage my config not on the server, but on my laptop. One of the cool tricks of nixos-rebuild is that you can deploy to a remote machine over SSH. This isn't even specific to flakes!

nixos-rebuild switch \
  --flake '.#gerbil' \
  --target-host gerbil \
  --use-remote-sudo

Here, --target-host can be anything you can SSH too. You can also add a username if necessary, but for these examples, I assume there's an alias gerbil in your ~/.ssh/config.

The above works if you are on Linux (more generally: Nix on your machine knows how to build for the target). But what if you're on, say, macOS? You can tell it to build on a remote machine with --build-host:

# If you don't have 'nixos-rebuild', for example on macOS.
nix-shell -p nixos-rebuild

nixos-rebuild switch \
  --flake '.#gerbil' \
  --build-host gerbil \
  --target-host gerbil \
  --use-remote-sudo

It's smart enough to detect when both the build and target hosts are the same, and avoids copying.

I personally put a wrapper with these flags in the devShell of my flake:

devShells = builtins.mapAttrs (system: pkgs:
let
  rebuild = pkgs.writeShellApplication {
    name = "gerbil-rebuild";
    runtimeInputs = [ pkgs.nixos-rebuild ];
    text = ''
      exec nixos-rebuild \
        --flake '.#gerbil' \
        --build-host gerbil \
        --target-host gerbil \
        --use-remote-sudo \
        "$@"
    '';
  };
in {
  default = pkgs.mkShell {
    nativeBuildInputs = [ rebuild ];
  };
}) nixpkgs.legacyPackages;

This allows you to do:

nix develop
gerbil-rebuild switch

But I can also highly recommend checking out direnv, which can give you nix develop functionality in your regular shell, as you cd into the directory. Magic!

Fixing automatic upgrades again

So you've made some changes, done gerbil-rebuild switch, then the next morning, your changes are gone. What the heck!

As it turns out, system.autoUpgrade.flake still points to the old version on the server itself. In order to do auto-upgrades, the server needs to have a current version of the configuration on-hand.

Rsync to the rescue

An easy solution is to keep a copy on the server. Keeping this in the home directory of your normal (non-root) user account is perfectly fine.

rsync --recursive --delete ./ gerbil:gerbil-config/

The --delete flag cleans up any files that shouldn't exist on the destination. Also mind that trailing / in paths is important for rsync!

Now change your auto-upgrade configuration to point to the new directory:

system.autoUpgrade = {
  enable = true;
  flake = "/home/stephan/gerbil-config#gerbil";
  flags = [ "--update-input" "nixpkgs" ];
};

If you end up sticking with this solution, you can create another convenient shell wrapper in your flake devShells for the rsync command.

Git bare clone on the server

Of course, you should be keeping your config in version control. With Git, you can keep a 'bare' clone on the server that you push to.

A bare repository is essentially just the .git directory, and no files / working tree. Not having a working tree avoids some issues when you push over SSH. (Git can't update a remote working tree, and will by default error.)

# On the server:
git init --bare gerbil-config

# Locally:
git remote add gerbil gerbil:gerbil-config
git push gerbil main

Now change your auto-upgrade configuration:

system.autoUpgrade = {
  enable = true;
  flake = "git+file:///home/stephan/gerbil-config";
  flags = [
    "--no-write-lock-file"
    "--update-input" "nixpkgs"
  ];
};

You can see flake is now a URL. This is because Nix supports bare Git repositories just fine, but can't detect them automatically, so we need to be explicit.

The extra --no-write-lock-file is necessary here, or Nix will hard error when it tries to update the input, because it cannot write flake.lock in a bare repository.

Using hosted Git

You can, of course, just use GitHub or some other hosted Git service:

system.autoUpgrade = {
  enable = true;
  flake = "github:stephank/gerbil-config#gerbil";
  flags = [
    "--no-write-lock-file"
    "--update-input" "nixpkgs"
  ];
};

Here too we need --no-write-lock-file, because Nix cannot write flake.lock to a remote flake.

If the repository is private, you will have to configure credentials to access it. You can do this via the access-tokens (for GitHub or Gitlab) or netrc-file options in nix.conf.

Note that auto-upgrades run as root, and you probably want to limit access to those credentials to the root user. To do that, manually place them in the root user's personal config file:
/root/.config/nix/nix.conf

Host your own Git

Because I have other projects that I don't want on a hosted Git service, but do want as flake inputs, I decided to build a dirt simple Git hosting myself. The only other option is to use git+file: inputs with absolute paths that are somehow the exact same on my laptop and the server. Not impossible, but a little insane.

Fortunately, setting up Git hosting is not too far fetched. Git comes bundled with a CGI executable that can help you serve over HTTPS.

If you're using Nginx, like me, you first need to hack around the fact Nginx doesn't do CGI. I don't really blame them; process management is a finicky part of Unix. There's a tool called fcgiwrap that translates FastCGI to CGI:

services.fcgiwrap = {
  enable = true;
  user = "nginx";
  group = "nginx";
};

With that out of the way, let's set up an Nginx virtual host:

{ config, pkgs, ... }:
let
  vhost = "git.stephank.nl";
in
{
  services.nginx.virtualHosts.${vhost} = {
    # Configure HTTPS using a Let's Encrypt certificate.
    enableACME = true;
    # Force HTTP -> HTTPS upgrade.
    forceSSL = true;

    locations."/" = {
      # Proxy to fcgiwrap.
      extraConfig = ''
        fastcgi_pass unix:${config.services.fcgiwrap.socketAddress};
      '';
      fastcgiParams = {
        # Tell fcgiwrap what CGI executable to run.
        SCRIPT_FILENAME = "${pkgs.git}/libexec/git-core/git-http-backend";
        # Required by git-http-backend.
        PATH_INFO = "$uri";
        # Serve everything underneath this directory.
        GIT_PROJECT_ROOT = "/srv/git";
        GIT_HTTP_EXPORT_ALL = "";
      };
    };

    # Access control.
    extraConfig = ''
      # Allow all traffic from the loopback interface.
      allow 127.0.0.2/32;
      allow ::1/128;
      # Add your home IPs here.
      allow 1.2.3.4/32;
      # Show everyone else the door.
      deny all;
    '';
  };

  # This is so that, when the server fetches from its own Git hosting, it uses
  # the loopback interface, and thus the 'allow' rules match.
  networking.hosts = {
    "127.0.0.2" = [ vhost ];
    "::1" = [ vhost ];
  };
}

So, I moved everything from my home directory to /srv/git, but my normal user account is still owner of everything inside, allowing me to push:

# On the server:
sudo mkdir -p /srv/git
sudo chown stephan:root /srv/git
mv gerbil-config /srv/git/

# Locally:
git remote set-url gerbil gerbil:/srv/git/gerbil-config

Now change your auto-upgrade configuration:

system.autoUpgrade = {
  enable = true;
  flake = "git+https://git.stephank.nl/gerbil-config#gerbil";
  flags = [
    "--no-write-lock-file"
    "--update-input" "nixpkgs"
  ];
};

Again, we need --no-write-lock-file, because Nix cannot write flake.lock to a remote flake.

Workflow

In practice, doing a remote deploy with nixos-rebuild in this setup is only useful to iterate on changes. When you're happy with the result, you must not forget to also commit and push, or auto-upgrades will trample over your changes.