Creating a Nix Package from a Python Project

Packing a Python project into a NixOS package is pretty tricky. There are a few things that you need to know to get it to work. I’ll walk you through how I packaged a Python project of mine.

My project is a collection of scripts that I needed to run on a recurring basis. This post does not cover scheduling the jobs on NixOS; it only goes over packaging the Python project as a Nix package. I wrote about scheduling jobs on NixOS using systemd in another post.

Python Project Setup

One thing I didn’t know when setting up this project was how difficult it would be to package a Python project. There are a ton of ways to do it.

I found a way to do it and just stuck with it. Once you figure out how to package your Python project, the NixOS packaging process is the same.

Python Project Basic Structure

First, we’re going to start with our Python project structure. My project has the directory structure below. The scripts directory contains the files that we actually call and execute, while py-library contains all of our tools and code that the scripts use.

├── scripts/                      # Contains the scripts that we'll execute
├── py-library/                   # Library of common functions
├── setup.py                      # This declares how our library is packaged
└── requirements.txt              # Helpful for pinning versions.

Defining Our Python Project

Since there are so many ways to package a Python script, I just went ahead and picked one. I settled for the setup.py and setuptools method. Below is my setup.py file definition:

setup.py

from setuptools import setup, find_packages
setup(
    name='py-library',
    version='0.0.1',
    install_requires=[
        'jsonpath-ng',
        'requests',
        'gspread',
        'duckdb; python_version == "3.11"',
    ],
    packages=find_packages(),
    scripts=[
        'scripts/script.py',
    ],
)

Note that there is an explicit mention of the scripts that are callable. Do not skip this step. I spent a bunch of time debugging this issue because my script was not defined here.

Remote Repository? No Problem

I like to store my projects in GitHub. Luckily the Nix language and builder allows us to build packages from a remote registry. There are a few things we need to start using it in our Nix package file

1. Repository Name and User The url should look something like the following: https://github.com/{user}/{repo}

2. GitHub Personal Access Token(PAT) If your repo is private, you need some way to authenticate against it when the machine attempts to pull and build the package. GitHub has a number of authentication tools avaliable. I found the easiest for this was the PAT. Go ahead and create one for your account that has access to the repository that contains your Python package.

Complete Nix Configuration

This is the complete Nix configuration for the Python project. You can go ahead and replace the variable names with those for your project.

# Below, we can supply defaults for the function arguments to make the script
# runnable with `nix-build` without having to supply arguments manually.
# Also, this lets me build with Python 3.11 by default but makes it easy
# to change the Python version for customized builds (e.g., testing).
{ nixpkgs ? import <nixpkgs> {}, pythonPkgs ? nixpkgs.pkgs.python311Packages }:

let
  # This takes all Nix packages into this scope.
  inherit (nixpkgs) pkgs;
  # This takes all Python packages from the selected version into this scope.
  inherit pythonPkgs;

  # Inject dependencies into the build function.
  f = { buildPythonPackage, requests, gspread, duckdb, jsonpath-ng }:
    buildPythonPackage rec {
      pname = "py-library";
      version = "0.0.1";
      
      # Pull source from a Git server. Optionally select a specific `ref` (e.g., branch),
      # or `rev` revision hash.
      src = builtins.fetchGit {
        url = "https://{github_pat}@github.com/{user}/{repo}.git";
        ref = "master";
        rev = "{your-commit_string}";
      };

      # Specify runtime dependencies for the package.
      propagatedBuildInputs = [
        requests
        gspread
        duckdb
        jsonpath-ng
      ];

      # If no `checkPhase` is specified, `python setup.py test` is executed
      # by default as long as `doCheck` is true (the default).
      # I want to run my tests in a different way:
      #  checkPhase = ''
      #    python -m unittest tests/*.py
      #  '';
      doCheck = false;

      # Meta information for the package.
      meta = {
        description = ''
          My Python project
        '';
      };
    };

  drv = pythonPkgs.callPackage f {};
in
  if pkgs.lib.inNixShell then drv.env else drv

Deploying and Starting

Once you get a working Python configuration for Nix, it’s very easy to use. One thing to note is that the Nix config file must be updated when your code changes, especially with something like dependencies.

Overall, I have mixed feelings about it. Now that I have it running, I’m going to continue using it. However, I think my time could have been spent better had I just thrown everything into a container and deployed it like that. Hopefully, this helps you if you need to write your own Nix package so you spend less time fumbling around than I did.