VPNs and re-routing systemd services

Networking is near and dear to my heart and where I started out professionally. It's exciting to be able to get back to it for whatever purpose I might need.

There are more VPN providers out there than you can shake a stick at and only slightly fewer ways to connect to them. They may have their own dedicated app (eww) or may support an open tool like OpenVPN or Wireguard. Those providers, they are great. This opens the door to other software tools that are capable of hooking up to whatever provider you may want to use.

I have a slightly different use case. I have a headless server that runs several different services. It may as well be named kitchen sink because of all the different things it runs. Some of those services, well, I'd prefer if those didn't come from my IP address. It's easier that way.

One of the more interesting tools I've come across is called vopono and it's different than other VPN clients because it uses a feature of the linux kernel called Network Namespaces. When it connects to a provider, it runs a command in a temporary namespace that's available only to that command so it doesn't hijack your system and redirect all traffic. Granted, you could rig this all up by hand without vopono, but let's stand on the shoulder's of giants.

I'd like to run one tunnel with a single provider but it would be most convenient if I could have multiple services running in the protected namespace. vopono's main invocation runs a single command but with some systemd tweaks, a single command is all you need.

There's no vopono service in nixpkgs. That makes some sense because it's not geared to being a system level service but that's not going to stop me. To make this happen though, I'll write a service so I can re-use code. It will need to do a few things:

• select a provider
• select a protocol
• select a server from that provider
• choose the default interface for outgoing traffic. if you have only one interface on your machine, it will auto detect properly.
• define services and ports and abduct them into the appropriate namespace.
• make it easy to run vopono sync. There isn't a way to set flags or variables to run an automated sync so let's wrap a command to make it possible to start up the service.

And here it is.

{ config, pkgs, lib, ... }:
with lib;
let
  cfg = config.services.vopono;
  inherit (builtins) listToAttrs attrValues toString;
  syncScript = pkgs.writeScriptBin "vopono-sync" ''
    set -eo pipefail
     [ -z "$SUDO_USER" ] && (echo "run with sudo"; exit 1)
    unset SUDO_USER
    vopono sync -c ${cfg.protocol} ${cfg.provider}
  '';
in
{

  options = {
    services.vopono = {
      enable = mkEnableOption "vopono service";

      provider = mkOption {
        description = "See vopono docs for valid providers.";
        type = types.str;
        example = "Mullvad";
      };
      protocol = mkOption {
        description = "See vopono docs for valid protocols.";
        type = types.str;
        example = "Wireguard";
      };
      server = mkOption {
        description = "See vopono docs for valid servers.";
        type = types.str;
        example = "usa-us";
      };
      package = mkOption {
        description = "The vopono package to use";
        type = types.package;
        default = pkgs.vopono;
      };
      interface = mkOption {
        type = types.str;
        default = "";
        description = "Optionally define the default interface. If not set, it uses the first interface on the system.";
      };
      namespace = mkOption {
        type = types.str;
        default = "sys_vo";
        example = "vopono";
        description = "Override the default, auto generated, namespace.";
      };

      services = mkOption {
        default = { };
        type = types.attrs;
        description = ''An attribute set with the name of a service where the value is a list of ports to forward to it.'';
        example = literalExpression ''
          { privoxy = [ 8118 ]; }
        '';
      };
    };
  };

  config =
    let
      /*
      Tie each service to vopono. If vopono stops/starts, these services will as well.
      Also bind in the resolv.conf for working DNS service and set the namespace for the service.
      */
      serviceReqs = {
        after = [ "vopono.service" ];
        partOf = [ "vopono.service" ];
        wantedBy = [ "vopono.service" ];
        serviceConfig = {
          BindPaths = [ "/etc/netns/${cfg.namespace}/resolv.conf:/etc/resolv.conf" ];
          NetworkNamespacePath = "/var/run/netns/${cfg.namespace}";
        };
      };
    in
    mkMerge [
      (mkIf cfg.enable {
        environment.systemPackages = [ syncScript ];
        systemd.services.vopono = {
          wantedBy = [ "multi-user.target" ];
          after = [ "network-online.target" ];
          path = with pkgs; [
            cfg.package
            wireguard-tools
            iproute2
            iptables
            procps
            systemd
            sudo
          ];
          unitConfig = {
            # Only run if a sync has been performed.
            ConditionPathExists = "/root/.config/vopono";
          };
          serviceConfig =
            let
              ports = unique (flatten (attrValues cfg.services));
              mkPorts = concatMapStrings (x: " -f ${toString x}") ports;
              interface = if cfg.interface != "" then "-i ${cfg.interface}" else "";
            in
            {
              # Wait for the service to announce itself as up via systemd-notify before carrying on
              Type = "notify";
              NotifyAccess = "all";
              Restart = "always";
              RestartSec = "5s";
              # This is running as root, and it includes sudo access, because there are some weird interactions if ran as a regular user.
              # From what I can tell, it drops permissions to run the single command and that's not interesting to us here.
              ExecStart = ''
                ${cfg.package}/bin/vopono exec ${interface} -u root --keep-alive ${mkPorts} --provider ${cfg.provider} --server ${cfg.server} --custom-netns-name ${cfg.namespace} --protocol ${cfg.protocol} "systemd-notify --ready"'';
              # It fails to start if there's a device left over from the last time it ran, purge it on stop.
              ExecStop = "${pkgs.iproute2}/bin/ip link delete ${cfg.namespace}_d";
            };
        };
      })
      { systemd.services = listToAttrs (map (x: { name = x; value = serviceReqs; }) (attrNames cfg.services)); }
    ];
}

And to set it up in my configuration.nix:

  imports = [ ./vopono.nix ];
  services.vopono = {
    enable = true;
    services = {
      privoxy = [8118];
    };
    protocol = "Wireguard";
    provider = "Mullvad";
    server =   "usa-us";
    interface = "enp8s0";
  };

Once the config lands and you have ran vopono-sync, you should get all of this good stuff from journald:

# journalctl -fu vopono
Jun 07 17:08:05 server vopono[2944915]:  2023-06-07T21:08:05.679Z INFO  vopono_core::util > Chosen config: /root/.config/vopono/mv/wireguard/usa-uslax401.conf
Jun 07 17:08:05 server vopono[2944915]:  2023-06-07T21:08:05.683Z INFO  vopono_core::network::netns > Created new network namespace: sys_vo
Jun 07 17:08:05 server vopono[2944915]:  2023-06-07T21:08:05.736Z INFO  vopono_core::network::netns > IP address of namespace as seen from host: 10.200.1.2
Jun 07 17:08:05 server vopono[2944915]:  2023-06-07T21:08:05.736Z INFO  vopono_core::network::netns > IP address of host as seen from namespace: 10.200.1.1
Jun 07 17:08:05 server vopono[2944915]:  2023-06-07T21:08:05.952Z INFO  vopono::exec                > Application systemd-notify --ready launched in network namespace sys_vo with pid 2945281
Jun 07 17:08:05 server sudo[2945281]:     root : PWD=/ ; USER=root ; COMMAND=/nix/store/75wxj2a3c0pdbf46bzmff8qr9vbjm5y1-systemd-253.3/bin/systemd-notify --ready
Jun 07 17:08:05 server sudo[2945281]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jun 07 17:08:05 server systemd[1]: Started vopono.service.
Jun 07 17:08:05 server sudo[2945281]: pam_unix(sudo:session): session closed for user root
Jun 07 17:08:06 server vopono[2944915]:  2023-06-07T21:08:06.034Z INFO  vopono::exec                > Keep-alive flag active - will leave network namespace alive until ctrl+C received

It takes advantage of systemd-notify to better communicate with systemd on when the service is up and responsive. Only after has the namespace has been configured and the tunnel is complete will systemd move on and start up the associated services. If defined, the ports for each services will be forwarded from the main namespace into tunnel namespace.

Let's also take a look at the service that has been co-opted into this namespace.

# systemctl cat privoxy.service | grep sys_vo
BindPaths=/etc/netns/sys_vo/resolv.conf:/etc/resolv.conf
NetworkNamespacePath=/var/run/netns/sys_vo

Do we have a functional resolv.conf that's different than the system?

# cat /etc/netns/sys_vo/resolv.conf
nameserver 193.138.218.74

Yep. Does traffic really work?

# ip netns exec sys_vo  curl -s ifconfig.co/json | jq .region_name
"Illinois"

# curl --proxy http://localhost:8118 -s ifconfig.co/json | jq .region_name
"Illinois"

I'm definitely not in Illinois. I have a proxy that I can point to for any random application and if I wanted to launch an app directly into the namespace on a whim, ip netns exec can do that for me too.

The service has some quality of life features as well for initial setup. If started without having an initial sync or config, it will spam the logs trying to run interactively. Using ConditionPathExists keeps this noise down and the service can only be started once it has a valid config.

So there we go. Some network namespacing with some systemd sprinkled in. Go forth and be anonymous.¹