hyper local automatic nix cache

the set up

I am an impatient pack rat. I hate waiting around for things and it bothers me to throw things away ¹ . I’ve got a wide number of projects that may go a long time without me touching them and they fall prey to the nix garbage collector.

A Nix GC would normally bug me due to my digital hoarding proclivity but I’ve figured out how to one up myself by being more efficient with what I keep around. Nix has a nifty caching mechanism which is able to take a store path (or the entire store) and bundle it into a nix archive (NAR) file. These files are fully self contained, much smaller than their sources, and ready to be re-used whenever you want them. I use them as a cold storage tier.

I also run a few NixOS machines. One of them has a bunch of storage on it and where I do a lot of my server-ish computing. They share some configs and consequently, would be able to save a bunch of CPU time by not having to re-build my custom stuff. Of course, there are other excellent tools for your own caching like Cachix and it’s an excellent product. My upload bandwidth is not particularly zesty though and few would ever want to take advantage of my own cache, so it’s not a great fit for my use case.

The interesting part about the NAR files though is that once you copy a path via nix copy --to "file:///srv/www/cache?compression=zstd" $some_path, you can plumb that directory to a web server and you have your own HTTP accessible nix store. Other machines could use it, or, follow along with me, how about just localhost?

I GC regularly now because everything my machine builds is automatically inserted into the cache. I have it clean up old files after a year and the working set is about 500gigs. If I GC’d a project that made it to the nix store, I save those precious moments when I come back to it that otherwise might fall victim to my undiagnosed ADHD.

how to do it

Here’s the appropriate config snippets.

{ config, pkgs, ... }:
let
  postBuildHook = pkgs.writeScript "nix-copy-paths" ''
    export IFS=' '
    mkdir -p /nix/var/nix/gcroots/nixcache
    for path in $OUT_PATHS; do
      nix-store --add-root /nix/var/nix/gcroots/nixcache/$(basename $path) -r $path
    done
  '';
in
{
  nix.extraOptions = ''
    # Add builds to a specific gcroots directory for nixcache to copy to.
    post-build-hook = ${postBuildHook}
    secret-key-files = /keys/nix_secret_signing_key
  '';
  nix.gc = {
    automatic = true;
    options = "--delete-older-than 60d";
  };
  nix.settings = {
   auto-optimise-store = false;
   keep-outputs = true;
   keep-derivations = true;
   substituters = [ "https://myhost/cache" ];
   trusted-public-keys = [ "mykey:u+Hmral+Ufo+TMLr4PMzQ2+rsvHFxOpor1rzRZhdQOQ=" ];
 };

  systemd.services.nixcache-clean = {
    startAt = "07:00:00";
    # I bet this could be converted to a tmpfiles.d config with a cleanup-age.
    serviceConfig.ExecStart =
      "${pkgs.findutils}/bin/find /srv/www/cache -atime 365 -delete";
  };

  systemd.services.nixcache = {
    path = with pkgs; [ config.nix.package coreutils inotify-tools ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig.Restart = "always";
    script = ''
      function arc {
        local path=$1
        shift
        npath=$(nix path-info -r /nix/store/$(basename $path))
        nix copy --to "file:///srv/www/cache?compression=zstd" $npath
        rm $path
        echo "copied $path"
      }
      # Catch any builds that might have accumulated while the service was stopped.
      for path in $(find /nix/var/nix/gcroots/nixcache/ -type l); do
        arc "$path"
        done
      # Continually watch the directory for new builds and copy them as they come.
      inotifywait -m -r -P -e moved_to  /nix/var/nix/gcroots/nixcache --format %w%f | while read path; do
        arc "$path"
      done
    '';
  };
  services.nginx = {
    virtualHosts."myhost" = {
      default = true;
      enableACME = true;
      root = "/srv/www";
    };
  };

}

At a high level, this snippet will:

Sign any build it creates with a secret key to make it easier for other machines to ingest.
Run postBuildHook on every build the nix-daemon does, adding a GC root to another directory.
Run a service watches that directory and copies every path it finds to the cache and then deletes it.
Run Nginx with a root directory to the cache.
Sets itself as a binary cache.

For bonus points, take note that there’s now a /srv/www/cache/nix-cache-info. The priority for cache.nixos.org is 30. If you make it lower, this cache will be preferred.

You might ask why I use a separate service rather than doing the copy in the postBuildHook. As I mentioned up top, I’m impatient. The build hook is a blocking call so if you have a large store path, it’s going to take a bit before it will return.

Check out the nixos wiki binary cache page for more background about operating a cache.

Footnotes

But I’m trying.