Setting up a container registry on NixOS

NixOS provides packages out of the box that work fairly well. One that I use on a daily basis is the Docker Registry package. Today I’ll walk through how I set this up for production usage.

Background

I’ve done a deep dive into NixOS and migrated all of my services to a NixOS and container-based solution. One of the first things I needed to do was set up a container registry to upload all of my containers. It was straightforward to get started, but there are some details that required quite a bit of debugging to figure out.

1. Starting up our registry

The dockerRegistry is a service built into NixPkgs. We can just go ahead and enable it:

services.dockerRegistry = {
    enable = true;
};

This starts the registry on localhost:5000 or http://127.0.0.1:5000. It’s not exposed to the outside world just yet.

2. Setting Up Nginx

Since our registry is not exposed to the public internet, we need a way of exposing it. The best way of handling this is by setting up a reverse proxy web server, as it provides features that we will use later.

1. Enabling Nginx

Like the Docker Registry, the Nginx package is included in Nix packages. We just need to enable it:

  services.nginx = {
      enable = true;
  };

2. Adding our VirtualHost Okay, so our webserver is enabled, but it isn’t doing anything. We need to configure it as a reverse proxy to allow a connection to our registry. We do this by adding a virtualHost configuration:

  services.nginx = {
    enable = true;
    virtualHosts = {
      "mywebsite.com" = {
        serverName = "mywebsite.com";
        locations."/" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:5000";
        };
      };
    };
  };

3. Basic Authentication & Allowing for Large Images

We’re going to add two more configurations here. First is our clientMaxBodySize configuration. This allows us to push large requests through the webserver. Container images are large blobs of data, so this is pretty much required.

Secondly, we’re going to add some authentication to our server. This ensures that only we, as owners of our registry, can push and pull images—not everyone on the internet. This is done with the basicAuth configuration in our virtual host:

  services.nginx = {
    enable = true;
    clientMaxBodySize = "2000m";
    virtualHosts = {
      "mywebsite.com" = {
        basicAuth = { username = "password";};
        locations."/" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:5000";
        };
      };
    };
  };

3. Certs on Certs

Unfortunately, Docker does not play well with registries that do not provide TLS or SSL encryption. You’ll get errors when you try to push or pull images to and from this registry. Luckily, NixOS provides TLS cert support via security.acme.

Once configured correctly, it handles initial generation and renewal of the cert via Let’s Encrypt. The tricky part, as with most things in NixOS, is getting the correct configuration.

1. Creating Our Security Config

Here we define some basic configurations for Let’s Encrypt, including the website name and a default email:

  security.acme = {
    acceptTerms = true;
    defaults.email = "myemail@email.com";
    certs = {
      "mywebsite.com" = {
        webroot = "/var/lib/acme/challenges-my-website";
        email = "myemail@email.com";
        group = "nginx";
      };
    };
  };

2. Creating our ACME User

For Nginx to automatically fetch the cert, we need to add a group to our users config:

  users.users.nginx.extraGroups = [ "acme" ];

3. Adding Cert to our VirtualHost

We need to modify our virtual host to enable TLS/SSL encryption. We wire up our Nginx configs to match our ACME configs:

  services.nginx = {
    enable = true;
    clientMaxBodySize = "2000m";
    virtualHosts = {
      "mywebsite.com" = {
        serverName = "mywebsite.com";
        useACMEHost = "mywebsite.com";
        forceSSL = true;
        acmeRoot = "/var/lib/acme/challenges-my-website";
        basicAuth = { username = "password";};
        locations."/" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:5000";
        };
      };
    };
  };

4. Complete Config

Now that we’ve got our security ducks in a row, let’s combine all the pieces we talked about. Your complete config should look similar to the following:

  services.dockerRegistry = {
    enable = true;
    enableGarbageCollect = true;
    garbageCollectDates = "monthly";
  };

  users.users.nginx.extraGroups = [ "acme" ];

  security.acme = {
    acceptTerms = true;
    defaults.email = "myemail@email.com";
    # Staging server for testing configurations....
    #defaults.server = "https://acme-staging-v02.api.letsencrypt.org/directory";
    certs = {
      "mywebsite.com" = {
        webroot = "/var/lib/acme/challenges-my-website";
        email = "myemail@email.com";
        group = "nginx";
      };
    };
  };

  services.nginx = {
    enable = true;
    clientMaxBodySize = "2000m";
    virtualHosts = {
      "mywebsite.com" = {
        serverName = "mywebsite.com";
        useACMEHost = "mywebsite.com";
        forceSSL = true;
        acmeRoot = "/var/lib/acme/challenges-my-website";
        basicAuth = { username = "password";};
        locations."/" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:5000";
        };
      };
    };
  };

What’s Next

Now that you’ve set up a container registry, setting up the rest of Nix to use containers is the easy part. Check out my guide on using your NixOS host for orchestrating containers.