Hacking on a package from nixpkgs

I fielded a question from the NixOS matrix channel and I thought it would be useful to expand on the response I gave.

There's an application in the nix package repository: https://github.com/NixOS/nixpkgs/tree/master/ I would like to modify the default.nix file. After I do that, how would I go about using it? Ideally in a nix-shell

There's a wild amount of flexibility that Nix and nixpkgs allows you to tackle this. Some of the fancier methods involve overlays or cloning nixpkgs and modifying your fork. Let's assume those don't exist and you aren't interested in cloning all of nixpkgs just for your single update. The way I would go about this doesn't involve an overlay up front but there's nothing stopping you from continuing on and using one if it fits your use case.

Let's take an example from a file that I hacked on myself for testing, pmacct. It's possible that it could go back upstream to nixpkgs but I wanted to test it locally first to see if it's worth it.

pmacct is a tool that collects network data, mutates it, and passes it on to some other process or data store. One optional dependency that nixpkgs doesn't include is the ndpi library. ndpi allows for Deep Packet Inspection to guess what kind of traffic is being passed. You may assume that a UDP53 flow is DNS, but it could be someone running a wireguard tunnel instead to evade firewalls. ndpi is already packaged, so I bet this will be easy.

Here's what we are working with in pkgs/tools/networking/pmacct/default.nix:

# pkgs/tools/networking/pmacct/default.nix
{ lib
, stdenv
, fetchFromGitHub
, pkg-config
, autoreconfHook
, libtool
, libpcap
, libcdada
# Optional Dependencies
, withJansson ? true, jansson
, withNflog ? true, libnetfilter_log
, withSQLite ? true, sqlite
, withPgSQL ? true, postgresql
, withMysql ? true, libmysqlclient, zlib, numactl
, gnutlsSupport ? false, gnutls
, testers
, pmacct
}:

stdenv.mkDerivation rec {
  version = "1.7.8";
  pname = "pmacct";

  src = fetchFromGitHub {
    owner = "pmacct";
    repo = "pmacct";
    rev = "v${version}";
    sha256 = "sha256-AcgZ5/8d1U/zGs4QeOkgkZS7ttCW6gtUv/Xuf4O4VE0=";
  };

  nativeBuildInputs = [
    autoreconfHook
    pkg-config
    libtool
  ];
  buildInputs = [
    libcdada
    libpcap
  ] ++ lib.optional withJansson jansson
  ++ lib.optional withNflog libnetfilter_log
  ++ lib.optional withSQLite sqlite
  ++ lib.optional withPgSQL postgresql
  ++ lib.optionals withMysql [ libmysqlclient zlib numactl ]
  ++ lib.optional gnutlsSupport gnutls;

  MYSQL_CONFIG = lib.optionalString withMysql "${lib.getDev libmysqlclient}/bin/mysql_config";

  configureFlags = [
    "--with-pcap-includes=${libpcap}/include"
  ] ++ lib.optional withJansson "--enable-jansson"
  ++ lib.optional withNflog "--enable-nflog"
  ++ lib.optional withSQLite "--enable-sqlite3"
  ++ lib.optional withPgSQL "--enable-pgsql"
  ++ lib.optional withMysql "--enable-mysql"
  ++ lib.optional gnutlsSupport "--enable-gnutls";

  passthru.tests = {
    version = testers.testVersion { package = pmacct; command = "pmacct -V"; };
  };

  meta = with lib; {
    description = "A small set of multi-purpose passive network monitoring tools";
    longDescription = ''
      pmacct is a small set of multi-purpose passive network monitoring tools
      [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry]
    '';
    homepage = "http://www.pmacct.net/";
    changelog = "https://github.com/pmacct/pmacct/blob/v${version}/ChangeLog";
    license = licenses.gpl2;
    maintainers = with maintainers; [ _0x4A6F ];
    platforms = platforms.unix;
  };
}

It's a straightforward derivation with quite a few optional attributes that could be disabled or enabled so let's be a good neighbor and follow the pattern with our own optional addition. Based on the docs for pmacct, all that should be required is to add the --enable-ndpi flag and ensure that the ndpi library is available.

I'll copy that file to a new directory outside of nixpkgs and make the update.

# pmacct.nix
{ lib
, stdenv
, fetchFromGitHub
, pkg-config
, autoreconfHook
, libtool
, libpcap
, libcdada
# Optional Dependencies
, withJansson ? true, jansson
, withNflog ? true, libnetfilter_log
, withSQLite ? true, sqlite
, withPgSQL ? true, postgresql
, withMysql ? true, libmysqlclient, zlib, numactl
, withNdpi ? true, ndpi
, gnutlsSupport ? false, gnutls
, testers
, pmacct
}:

stdenv.mkDerivation rec {
  version = "1.7.8";
  pname = "pmacct";

  src = fetchFromGitHub {
    owner = "pmacct";
    repo = "pmacct";
    rev = "v${version}";
    sha256 = "sha256-AcgZ5/8d1U/zGs4QeOkgkZS7ttCW6gtUv/Xuf4O4VE0=";
  };

  nativeBuildInputs = [
    autoreconfHook
    pkg-config
    libtool
  ];
  buildInputs = [
    libcdada
    libpcap
  ] ++ lib.optional withJansson jansson
  ++ lib.optional withNflog libnetfilter_log
  ++ lib.optional withSQLite sqlite
  ++ lib.optional withPgSQL postgresql
  ++ lib.optionals withMysql [ libmysqlclient zlib numactl ]
  ++ lib.optional withNdpi ndpi
  ++ lib.optional gnutlsSupport gnutls;

  MYSQL_CONFIG = lib.optionalString withMysql "${lib.getDev libmysqlclient}/bin/mysql_config";

  configureFlags = [
    "--with-pcap-includes=${libpcap}/include"
  ] ++ lib.optional withJansson "--enable-jansson"
  ++ lib.optional withNflog "--enable-nflog"
  ++ lib.optional withSQLite "--enable-sqlite3"
  ++ lib.optional withPgSQL "--enable-pgsql"
  ++ lib.optional withMysql "--enable-mysql"
  ++ lib.optional withNdpi "--enable-ndpi"
  ++ lib.optional gnutlsSupport "--enable-gnutls";

  passthru.tests = {
    version = testers.testVersion { package = pmacct; command = "pmacct -V"; };
  };

  meta = with lib; {
    description = "A small set of multi-purpose passive network monitoring tools";
    longDescription = ''
      pmacct is a small set of multi-purpose passive network monitoring tools
      [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry]
    '';
    homepage = "http://www.pmacct.net/";
    changelog = "https://github.com/pmacct/pmacct/blob/v${version}/ChangeLog";
    license = licenses.gpl2;
    maintainers = with maintainers; [ _0x4A6F ];
    platforms = platforms.unix;
  };
}

In theory, that's a working package but how to use it? To flake or not to flake? That is the question. Either approach is perfectly fine and there are trade offs with each. I'll show both. I won't be pinning my nixpkgs version in code here so picking unstable or release channels will be an exercise to the reader. That's something I would recommend if going beyond testing. The flake example will have a bit more data as I will be using flake-parts because I enjoy it's features but that's as much scope creep as I'll add here.

For non-flakes:

# shell.nix
with import <nixpkgs> {};
let
  pmacct = pkgs.callPackage ./pmacct.nix {};
in
pkgs.mkShell {
  name = "pmacct-env";
  buildInputs = [ pmacct ];
}

Flake:

# flake.nix:
{
  description = "pmacct testing";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };
  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      imports = [];
      systems = [ "x86_64-linux" "aarch64-linux" ]; # no darwin support on this app.
      perSystem = { config, self', inputs', pkgs, system, ... }: {
        packages.default = pkgs.callPackage ./pmacct.nix {};
        devShells.default = pkgs.mkShell {
          name = "pmacct-env";
          buildInputs = [ self'.packages.default ];
        };
      };
      flake = {};
    };
}

And let's see what happens!

0ogre:~/pm% nix develop
warning: creating lock file '/home/cransom/pm/flake.lock'
error: builder for '/nix/store/d6sq7hki17xxw0d05fvwgdrmiii9gdw9-pmacct-1.7.8.drv' failed with exit code 2;
       last 10 log lines:
       > /nix/store/x69g1pv47g6kfynchhbr2w0x27zya83g-ndpi-4.6/include/ndpi/ndpi_api.h:323:17: note: declared here
       >   323 |   ndpi_protocol ndpi_detection_process_packet(struct ndpi_detection_module_struct *ndpi_struct,
       >       |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       > make[3]: *** [Makefile:430: ndpi.lo] Error 1
       > make[3]: Leaving directory '/build/source/src/ndpi'
       > make[2]: *** [Makefile:1346: all-recursive] Error 1
       > make[2]: Leaving directory '/build/source/src'
       > make[1]: *** [Makefile:699: all] Error 2
       > make[1]: Leaving directory '/build/source/src'
       > make: *** [Makefile:743: all-recursive] Error 1
       For full logs, run 'nix log /nix/store/d6sq7hki17xxw0d05fvwgdrmiii9gdw9-pmacct-1.7.8.drv'.
error: 1 dependencies of derivation '/nix/store/9x3b3pdk8jhsdf1fb43y8j6igmnqpqgc-pmacct-env-env.drv' failed to build

Oops. Same thing happens with nix-shell. Something seems off with our ndpi. Let's check the docs for pmact.

2. Download nDPI from its GitHub repository (https://github.com/ntop/nDPI). The nDPI API tends to change frequently in a non-backward compatible fashion. This means that not every version of pmacct will work with any version of nDPI. pmacct 1.7.8 works against nDPI 4.0-stable, 4.2-stable and 4.4-stable. pmacct 1.7.9 does work against 4.6-stable.

That could be a problem. What version of ndpi are we using? nix repl can tell us.

0ogre:~/pm% nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> :lf .
Added 16 variables.

nix-repl> inputs.nixpkgs.legacyPackages.x86_64-linux.ndpi.version
"4.6"

That makes sense, our pmacct is 1.7.8 and it tells us that it's not compatible. One more tweak then, lets bump our pmacct to the as-of-yet-unreleased, bleeding edge 1.7.9 because I'd like to have the newest awesomeness from ndpi.

# pmacct.nix
#<snipped for brevity >
stdenv.mkDerivation rec {
  version = "1.7.9pre";
  pname = "pmacct";

  src = fetchFromGitHub {
    owner = "pmacct";
    repo = "pmacct";
    rev = "8e09ee564d25716f60aed58e8b482bae96e6d31b";
    hash = "sha256-IV2C0NpLxGDXlC+gETkGGvv9QIB87lJP0Anu0f3yTa8=";
  };
};

#<snip for brevity>

I pulled this fetchFromGitHub invocation via nurl since it's a bit quicker than grabbing a rev and then using a fake hash to figure out what the real hash should be.

Now what do I have?

<build output snipped>
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/sfacctd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/pmacctd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/pmbgpd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/pmbmpd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/pmtelemetryd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/nfacctd
shrinking /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/bin/pmacct
checking for references to /build/ in /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre...
moving /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin/* to /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/bin
patching script interpreter paths in /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre
stripping (with command strip and flags -S -p) in  /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/lib /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/bin /nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre/sbin
running post-build-hook '/nix/store/9kgr417cacrq93jkqyvil67jhyxw1kaj-nix-copy-paths'...
post-build-hook: /nix/var/nix/gcroots/nixcache/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre

[nix-shell:~/pm]$ pmacct -V
pmacct IMT plugin client, pmacct 1.7.9-git (RELEASE)
 '--disable-static' '--disable-dependency-tracking' '--prefix=/nix/store/agbrwxgin17xlvcwhx6nvam35m441p8f-pmacct-1.7.9pre' '--with-pcap-includes=/nix/store/6kmiilv2nc6dixfmga7ij84jjzlm7zwb-libpcap-1.10.4/include' '--enable-jansson' '--enable-nflog' '--enable-sqlite3' '--enable-pgsql' '--enable-mysql' '--enable-ndpi' 'CC=gcc' 'PKG_CONFIG=pkg-config' 'PKG_CONFIG_PATH=/nix/store/8f3x783lrkh7c2wgr8xcy75k6kg2qdjp-file-5.44-dev/lib/pkgconfig:/nix/store/6kmiilv2nc6dixfmga7ij84jjzlm7zwb-libpcap-1.10.4/lib/pkgconfig:/nix/store/147l7602q8xpy3w30n843g8wx4hb9g8n-jansson-2.14/lib/pkgconfig:/nix/store/yxlx2w6q2i5s2vxhqhklacn3g1b3ab7w-libnetfilter_log-1.0.2/lib/pkgconfig:/nix/store/9lwyyxxqkwya33pq29hjs9f8pjfgch3g-libnfnetlink-1.0.2/lib/pkgconfig:/nix/store/pm89j4jal6mjhlr082rka8s0v0hm23y7-sqlite-3.41.2-dev/lib/pkgconfig:/nix/store/czrrap8sqkkplqj6j8b8r97jca7zfcbm-postgresql-14.8/lib/pkgconfig:/nix/store/qp6550ja9mw6nhwdv38mnczj4g0vam44-mariadb-connector-c-3.2.5-dev/lib/pkgconfig:/nix/store/nkfn58xv8nzp0r67izyhhmsr39v8ksjh-curl-8.0.1-dev/lib/pkgconfig:/nix/store/cvmxdaa0kx3w6gg819kqiqm3hcslak3j-brotli-1.0.9-dev/lib/pkgconfig:/nix/store/m9vssg00klfyyffi6qnvk0hgkd714vad-libkrb5-1.20.1-dev/lib/pkgconfig:/nix/store/ply36mibbhmrwxpan306kx8ag5v6kyr7-nghttp2-1.51.0-dev/lib/pkgconfig:/nix/store/a72gwwqm4k782ci0idylghqy4kwkgm1s-libidn2-2.3.4-dev/lib/pkgconfig:/nix/store/fd6fcsbbdhl8pbpcbfmyapkd66yxhcjd-openssl-3.0.8-dev/lib/pkgconfig:/nix/store/z299ddxdvq9innpr5g3qiw21qmmzvdx3-libssh2-1.10.0-dev/lib/pkgconfig:/nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/lib/pkgconfig:/nix/store/wa3biqwfmh7lq84qw9gwrwbw8rgrhf4y-zstd-1.5.5-dev/lib/pkgconfig:/nix/store/1zq3bi0xs3m3lsq4r4vlspx9vzygg2xk-numactl-2.0.16-dev/lib/pkgconfig:/nix/store/x69g1pv47g6kfynchhbr2w0x27zya83g-ndpi-4.6/lib/pkgconfig' 'CXX=g++' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'

For suggestions, critics, bugs, contact me: Paolo Lucente <paolo@pmacct.net>.

There we go, pmacct with ndpi support in my shell. With a little bit of copy-paste into a system config, it's available system wide.

That's it. If you want the full files, here they are:

• shell.nix
• flake.nix
• pmacct.nix