Nixpkgs support for Linux builders running on macOS

I recently upstreamed a derivation for a Linux builder into Nixpkgs that’s easy to deploy on macOS. The significance of this work is that you can now run the following command on macOS:

$ nix run nixpkgs#darwin.builder

… and that will launch a Linux builder that you can delegate builds to. For full details, read the corresponding section of the Nixpkgs manual.

In this post, I wanted to provide some of the background and motivation for this work to help contextualize it.

Background - NixOS qemu VMs on MacOS

I wasn’t originally trying to create a Linux builder for macOS when I began this project. I was actually working on making it as easy as possible to experiment interactively with (non-builder) NixOS qemu VMs on macOS.

While searching for prior art related to this I stumbled across the following Nixpkgs issue requesting exactly this same feature: Allowing NixOS VM’s to be run on macOS.

Even better, by the time I discovered that issue several people had already done most of the work, culminating in the following repository demonstrating how all of the features were supposed to fit together: YorikSar/nixos-vm-on-macos.

In fact, the flake for that repository also came with a binary cache, so if you just ran:

$ nix run github:YorikSar/nixos-vm-on-macos

… then you could run the sample NixOS VM from that repository on macOS without requiring access to an Linux builder because it would download all the Linux build products from the matching cache. Pretty neat!

However, this still didn’t completely satisfy my use case for reasons already noted by someone else: it doesn’t work well if you want to run a NixOS VM that differs even slightly from the included sample VM. Any difference requires Linux build products to be rebuilt which requires access to a Linux builder because those build products will not be cached ahead of time.

Background - linuxkit-nix

The need for a Linux builder wasn’t a showstopper for me because there was already prior art for bootstrapping a Linux builder on macOS, which was the linuxkit-nix project. So what I could have done was:

• Launch a (non-NixOS) linuxkit VM on macOS for use as a Linux builder
• Use the linuxkit builder to build the desired NixOS qemu VM
• Run that NixOS qemu VM on macOS

However, I was curious if I could use a NixOS VM for the first step, too! In other words:

• Launch a cached NixOS qemu VM on macOS for use as a Linux builder
• Use the qemu builder to build the desired (non-builder) NixOS qemu VM
• Run that NixOS qemu VM on macOS

The only difference between the two approaches is the first step: instead of using linuxkit to create the Linux builder we use qemu to create a NixOS builder. This works because the qemu builder’s NixOS configuration doesn’t need to change so hydra.nixos.org can build and cache the NixOS qemu builder ahead of time.

There were a few reasons I took interest in this approach:

• linuxkit-nix appears to not work on aarch64-darwin (i.e. Apple Silicon)

  This seems like it is potentially fixable, but I wasn’t yet ready to volunteer to do that work.

• It’s easier to customize a NixOS builder

  linuxkit-nix doesn’t use NixOS for the builder and instead creates a bespoke builder for this purpose. This means that you can’t use the NixOS module system to more easily customize the behavior of the builder.

• The qemu-based solution is simpler than linuxkit-nix

  I think the easiest way to explain this is for me to link to the macos-builder.nix NixOS module, which has the entirety of the code that I contributed, which is significantly simpler than linuxkit-nix.

  The main reason that the qemu-based solution is simpler than linuxkit-nix is because it is reusing more infrastructure that has already been upstreamed into Nixpkgs (most notably, NixOS and qemu VMs).

• linuxkit-nix appears to be unmaintained

  There was a nascent attempt to upstream linuxkit-nix into Nixpkgs, but that stalled because it seems like linuxkit-nix appears to have been abandoned a couple of years ago.

  I could have restored that effort, but personally I was fine with using the simpler qemu-based approach. I haven’t given up on the idea of reviving linuxkit-nix, but it’s not on my immediate roadmap.

There is one notable downside to using qemu over linuxkit, which is that qemu is supposed to be slower than linuxkit

Note: I have not actually verified this claim since I can’t run linuxkit-nix on my M1 Mac, but this is purportedly the reason that the authors of linuxkit-nix did not opt to use qemu for their approach according to this PR description.

qemu performance hasn’t been an issue for me (yet), but that could change, especially if I try to make use of this at work, where performance could potentially matter more.

Motivation

As I mentioned above, the long-term goal for all of this is to run NixOS VMs on macOS. There are two main reasons I’m interested in this:

• I’m working on a NixOS book

  … and I wanted macOS users to be able to test-drive example NixOS configurations on their local machine without requiring them to own and operate a separate Linux machine.

• I’m interested in running NixOS tests on macOS

  … primarily for work-related reasons. At work developers have to install postgres on their development machines for integration testing, and it would be much nicer if we could restructure our integration tests as NixOS tests (which run inside of qemu VMs instead of running on the host).

  However, at the time of this writing this would still require additional work which is in progress on this draft pull request.