org-howto/nix/nix-for-your-own-project.org

Building your own project with nix

• Motivation
• Lessons
  • Keep .nix files and project source in separate repos
    • Why not
    • Do this instead
  • Setup your build the nixpkgs way
  • Don't put build instructions in flakes
• Example Flake Setup
  • File cmake-examples-nix/pkgs/ex23.nix:
  • File cmake-examples-nix/flake.nix:
  • Explanation
    • List derivations to build
    • Build for host architecture
    • Overlay to choose configuration
    • Delegate build instructions and respect flake-mediated pinning
  • Use
  • Links

#

#

#

Motivation

I've found plenty of getting-started guides for working with nix. Most take the perpective of "a reproducible apt-get": they cover how to use nix to assemble/modify external software packages.

Less common is instruction on how best to package local (non-nixpkgs) software, in particular if you have multiple packages with non-trivial interdependencies.

The sequel summarizes my current understanding of how best to make this work (thanks to Jade Lovelace's article on flakes and nixpkgs, link below)

Lessons

Keep .nix files and project source in separate repos

On the face of it, it seems attractive and natural to keep nix files for a project in the same repo! That's what we do with build instructions (CMakeLists.txt etc)

Why not

1. .nix files (default.nix, shell.nix etc) likely need to contain revision hashes. Want to avoid situation where you've modified default.nix, and need it to record a hash that wil be determined… after you've committed the file to git
2. You might want to incorporate a repo child into another repo parent as a submodule. The .nix files associated with child probably don't do what you want in that context.
3. Even stronger reason to avoid this: nix flakes. Flakes will break (at least as of Dec 2023) if a directory that contains a flake.lock file contains another directory with a flake.lock file. The flake system can't exit from a state in which directory tree is dirty. This nixes composing project trees that contain flakes.
4. The nixpkgs authors have reached the same conclusion. In nixpkgs, instructions for building a package are expressed as a function, to be invoked from callPackage. The nix-pkgs provides for features like parameterization and cross-compilation; tradeoff is that they then don't work directly from 'nix-build'

Do this instead

For a project foo, create a foo-nix repo that uses fetchurl or cousins to fetch and build foo.

Setup your build the nixpkgs way

In other words, use callPackage!

1. callPackage supports cross-compilation (out of the box)
2. callPackage supports parameterization (i.e. overrides)
3. If you already have nix build setup for nixpkgs, then there's nothing new to do if you want to upload into nixpkgs (or incorporate into your fork thereof) later

Don't put build instructions in flakes

Do use flakes! They're great for version pinning and specifying configurations

However, if you put build instructions in flakes (i.e. call outside callPackage), then you lose the ability to easily compose and adapt that the nixpkgs tools give you.

For packaging our own software we will use flake to do these things (and nothing else):

1. pin nixpkgs.
2. fetch repos for local projects (things we aren't getting from nixpkgs)
3. specifying dependendency sets for local projects.

In particular we exclude from flake:

1. stdenv.mkDerivation calls
2. specifying buildInputs or cousins

Example Flake Setup

Here's an example, following advice above This example condensed from cmake-examples-nix. We have a local project cmake-examples, with a bunch of nixpkgs dependencies.

We'll setup a nix build in cmake-examples-nix. Will need just two files:

• flake.nix, a nix flake
• pkgs/ex23.nix, build instructions as in nixpgks.

File cmake-examples-nix/pkgs/ex23.nix:

  {
    # dependencies
    stdenv, doxygen, cmake, catch2, pkg-config, python3Packages, boost, zlib, # ... other deps here

    # args
    #   someconfigurationoption ? false
    pybind11 ? python3Packages.pybind11,
    sphinx ? python3Packages.sphinx,
    breathe ? python3Packages.breathe,
  } :

  stdenv.mkDerivation (finalattrs:
    {
      name = "cmake-examples-ex23";

      # note: ../flake.nix will override this
      src = fetchGit {
        url = "https://github.com/rconybea/cmake-examples";
        ref = "ex23";     # branch
        # rev = "12345";  # hash
        sha256 = "";      # must supply if not overridden from parent
      };

      # run unit tests
      doCheck = true;

      buildInputs = [ cmake pkg-config  pybind11 sphinx doxygen breathe catch2 boost zlib ];

      buildPhase = ''
        make
        make doxygen
      '';
    })

File cmake-examples-nix/flake.nix:

  {
    description = "Flake for cmake-examples project";

    # pinning nixpkgs
    inputs.nixpkgs.url = "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz";
    # inputs.nixpkgs.url = "github:nixos/nixpkgs/23.11";  # to use particular stable version

    inputs.flake-utils.url = "github:numtide/flake-utils";

    # branch 'ex23' from cmake-examples
    inputs.cmake-examples-ex23-path = { type = "github"; owner = "Rconybea"; repo = "cmake-examples"; flake = false; ref = "ex23"; };

    outputs = { self,
                nixpkgs,
                flake-utils,
                cmake-examples-ex23-path
              } :
                let
                  out = system :
                    let
                      pkgs = nixpkgs.legacyPackages.${system};
                      appliedOverlay = self.overlays.default pkgs pkgs;
                    in
                      {
                        # 1 line for each of our own packages
                        packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23;
                      };
                in
                  flake-utils.lib.eachDefaultSystem out // {
                    overlays.default = final: prev: (
                      let
                        # configuration choices
                        boost = prev.boost182;                     # boost 1.82
                        python3Packages = prev.python311Packages;  # python 3.11

                        # configuration choices we're making here
                        extras = { boost = boost; python3Packages = python3Packages; };
                      in
                        {
                          cmake-examples-23 =
                            (prev.callPackage ./pkgs/ex23.nix { boost = boost;
                                                                python3Packages = python3Packages; })
                              .overrideAttrs(old: { src = cmake-examples-ex23-path; });
                        });
                  };
    }

Explanation

Diving into some of the contents of these two .nix files:

List derivations to build

In flake.nix we have:

  let
    pkgs = nixpkgs.legacyPackages.${system};
    appliedOverlay = self.overlays.default pkgs pkgs;
  in
    {
      # 1 line for each of our own packages
      packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23;
    };

Here packages.cmake-examples-ex23 represents a derivation to be built by this flake.

Build for host architecture

  flake-utils.lib.eachDefaultSystem out // ...

establishes builds compatible with host architecture (not cross-compiling). out is target architecture. The // substitutes attributes from its RHS, discussed below:

Overlay to choose configuration

#+begin-src overlays.default = final: prev: (….);

#+end_src specifies an overlay, in "lazy-converging-to-a-fixpoint style" (which is apparently a thing).

When nix invokes overlays.default:

• prev refers to a guess (at entire altered-nixpkgs-expression?) from prior iteration.
• final refers to current iteration.

After first iteration, when prev and final are the same, nix:

• recognizes that it has reached a fixpoint,
• declares victory
• uses the derivations from either of the converged arguments (since they're the same).

The iteration is initiated by nixpkgs for selected architecture twice (as prev and final):

  pkgs = nixpkgs.legacyPackages.${system}
  appliedOverlay = self.overlays.default pkgs pkgs;

We make configuration choices here:

  boost = prev.boost182;
  python3Packages = prev.python311Packages;

If the LHS names matches something in nixpkgs, then that line can be omitted to adopt whatever default nixpkgs offers.

For example:

$ nix-env -qaP | grep nixpkgs.boost
nixpkgs.boost175    boost-1.75.0
nixpkgs.boost177    boost-1.77.0
nixpkgs.boost178    boost-1.78.0
nixpkgs.boost       boost-1.79.0
nixpkgs.boost180    boost-1.80.0
nixpkgs.boost181    boost-1.81.0
nixpkgs.boost182    boost-1.82.0

So our example configuration chooses boost-1.82.0 instead of nixpkgs default boost-1.79.0

Delegate build instructions and respect flake-mediated pinning

We finally introduce a derivation for our own package (cmake-examples-23) here:

    cmake-examples-23 =
      (prev.callPackage ./pkgs/ex23.nix { boost = boost;
                                          python3Packages = python3Packages; })
        .overrideAttrs(old: { src = cmake-examples-ex23-path; });

Here

    { boost = boost;
      python3Packages = python3Packages; }

introduces our overrides (we're using flake.nix for the approved purpose of making configuration choices) to arguments of the top-level function in ./pkgs/ex23.nix

Meanwhile

    .overrideAttrs(old: { src = cmake-examples-ex23-path; })

tells nix to override the src attribute in ./pkgs/ex23.nix's argument to stdenv.mkDerivation

In other words, it substitutes for:

      src = fetchGit {
        url = "https://github.com/rconybea/cmake-examples";
        ref = "ex23";     # branch
        # rev = "12345";  # hash
        sha256 = "";      # must supply if not overridden from parent
      };

Use

Use result just like a regular flake

Verify flake:

$ cd cmake-examples-nix
$ nix flake check

Example flake.lock:

  {
    "nodes": {
      "cmake-examples-ex23-path": {
        "flake": false,
        "locked": {
          "lastModified": 1709699013,
          "narHash": "sha256-Polpd2+DiZF615sih6HLOtzj0LaaFkQxrN6Jobdnq7M=",
          "owner": "Rconybea",
          "repo": "cmake-examples",
          "rev": "6d14f4146f2c7fc2011dd6790947ea261575304e",
          "type": "github"
        },
        "original": {
          "owner": "Rconybea",
          "ref": "ex23",
          "repo": "cmake-examples",
          "type": "github"
        }
      },
      "nixpkgs": {
        "locked": {
          "narHash": "sha256-mBXQ65IrCJbNgTrj0+6xdXpD9/U31AWPKdwGlOufhtI=",
          "type": "tarball",
          "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
        },
        "original": {
          "type": "tarball",
          "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
        }
      },
      ...
    },
    "root": "root",
    "version": 7
  }

Build (from pinned revisions)

$ nix build -L --print-build-logs .#cmake-examples-ex23

Result

$ tree ./result
result
├── 3.11
│   ├── pyzstream.cpython-311-x86_64-linux-gnu.so
│   └── zstream.py
├── bin
│   ├── hello
│   └── myzip
├── include
│   ├── compression
│   │   ├── base_zstream.hpp
│   │   ├── buffer.hpp
│   │   ├── buffered_deflate_zstream.hpp
│   │   ├── buffered_inflate_zstream.hpp
│   │   ├── compression.hpp
│   │   ├── deflate_zstream.hpp
│   │   ├── hex.hpp
│   │   ├── inflate_zstream.hpp
│   │   ├── span.hpp
│   │   └── tostr.hpp
│   └── zstream
│       ├── xfilebuf.hpp
│       ├── zstream.hpp
│       └── zstreambuf.hpp
├── lib
│   ├── libcompression.so -> libcompression.so.2.3
│   ├── libcompression.so.2
│   └── libcompression.so.2.3 -> libcompression.so.2
└── share
    └── doc
        └── cmake-examples
            └── html
                 ...

Links

Resources

• https://jade.fyi/blog/flakes-arent-real/ wonderful Jade Lovelace blog – inspiration for this article!
• https://github.com/vlktomas/nix-examples Tomas Vlk nix examples
• https://nixos.org/guides/nix-pills/ Nix pills (nix tutorial, from the ground up)
• https://ianthehenry.com/posts/how-to-learn-nix/ Ian Henry's "nix travel diary"
• https://ryantm.github.io/nixpkgs/stdenv/stdenv Nix standard environment docs
• https://github.com/Rconybea/cmake-examples my progressive series of cmake examples, using as example of a local project
• https://github.com/Rconybea/cmake-examples-nix nix build for cmake-examples