Recently I’ve been working on writing a “NixOS in Production”
book and one of the chapters I’m writing is on deploying NixOS using
terraform
. However, one of the issues I ran across was the
poor NixOS support for terraform
. I’ve already gone through
the nix.dev
post explaining how to use the terraform-nixos
project but I ran into several issues trying to follow those
instructions (which I’ll explain below). That plus the fact that
terraform-nixos
seems to be
unmaintained pushed me over the edge to rewrite the project to
simplify and improve upon it.
So this post is announcing my terraform-nixos-ng
project:
… which is a rewrite of terraform-nixos
and I’ll use
this post to compare and contrast the two projects. If you’re only
interested in trying out the terraform-nixos-ng
project
then go straight to the
README
Using nixos-rebuild
One of the first things I noticed when kicking the tires on
terraform-nixos
was that it was essentially reinventing
what the nixos-rebuild
tool already does. In fact, I was so
surprised by this that I wrote a standalone post explaining how to use
nixos-rebuild
as a deployment tool:
Simplifying that code using nixos-rebuild
fixed lots of
tiny papercuts I had with terraform-nixos
, like:
The deploy failing if you don’t have a new enough version of
bash
installedThe inability to turn off the use of the
--use-substitutes
flagThat flag causes issues if you want to deploy to a machine that disables outbound connections.
The dearth of useful options (compared to
nixos-rebuild
)… including the inability to fully customize
ssh
optionsThe poor interop with flakes
For example,
terraform-nixos
doesn’t respect the standardnixosConfigurations
flake output hierarchy.Also,
terraform-nixos
doesn’t use flakes natively (it usesflake-compat
), which breaks handling of theconfig.nix.binary{Caches,CachePublicKeys}
flakes settings. The Nix UX for flakes is supposed to ask the user to consent to those settings (because they are potentially insecure to auto-enable for a flake), but their workaround breaks that UX by automatically enabling those settings without the user’s consent.
I wanted to upstream this rewrite to use nixos-rebuild
into terraform-nixos
, but I gave up on that idea when I saw
that no pull request since 2021 had been merged, including conservative
pull requests like this
one to just use the script included within the repository to update
the list of available AMIs.
That brings me to the next improvement, which is:
Auto-generating available AMIs
The terraform-nixos
repository requires the AMI list to
be manually updated. The way you do this is to periodically run a script
to fetch the available AMIs from Nixpkgs and then create a PR to vendor
those changes. However, this shouldn’t be necessary because we could
easily program terraform
to generate the list of AMIs on
the fly.
This is what the terraform-nixos-ng
project does, where
the ami
module creates a data
source that runs an equivalent script to fetch the AMIs at
provisioning time.
In the course of rewriting the AMI module, I made another small improvement, which was:
Support for aarch64
AMIs
Another gripe I had with terraform-nixos-ng
is that its
AMI module doesn’t support aarch64-linux
NixOS AMIs even
though these AMIs exist and Nixpkgs
supports them. That was a small and easy fix, too.
Functionality regressions
terraform-nixos-ng
is not a strict improvement over
terraform-nixos
, though. Specifically, the most notable
feature omissions are:
Support for non-flake workflows
terraform-nixos-ng
requires the use of flakes and doesn’t provide support for non-flake-based workflows. I’m very much on team “Nix flakes are good and shouldn’t be treated as experimental any longer” so I made an opinionated choice to require users to use flakes rather than support their absence.This choice also isn’t completely aesthetic, the use of flakes improves interop with
nixos-rebuild
, where flakes are the most ergonomic way fornixos-rebuild
to select from one of many deployments.Support for secrets management
I felt that this should be handled by something like
sops-nix
rather than rolling yet another secrets management system that was idiosyncratic to this deploy tool. In general, I wanted theseterraform
modules to be as lightweight as possible by making more idiomatic use of the modern NixOS ecosystem.Support for Google Compute Engine images
terraform-nixos
supports GCE images and the only reason I didn’t add the same support is because I’ve never used Google Compute Engine so I didn’t have enough context to do a good rewrite, nor did I have the inclination to set up a GCE account just to test the rewrite. However, I’d accept a pull request adding this support from someone interested in this feature.
Conclusion
There’s one last improvement over the terraform-nixos
project, which is that I don’t leave projects in an abandoned state.
Anybody who has contributed to my open source projects knows that I’m
generous about handing out the commit bit and I’m also good about
relinquishing control if I don’t have time to maintain the project
myself.
However, I don’t expect this to be a difficult project to maintain
anyway because I designed terraform-nixos-ng
to outsource
the work to existing tools as much as possible instead of reinventing
the wheel. This is why the implementation of
terraform-nixos-ng
is significantly smaller than
terraform-nixos
.