Sunday, August 28, 2022

Incrementally package a Haskell program using Nix

incremental-nix

This post walks through how to take a standalone Haskell file and progressively package the file using Nix. In other words, we will tour a spectrum of packaging options ranging from simple to fancy.

The running example will be the following standalone single-file Haskell program:

I won’t go into detail about what that program does, although you can study the program if you are curious. Essentially, I’m planning to deliver a talk based on that program at this year’s MuniHac and I wanted to package it up so that other people could collaborate on the program with me during the hackathon.

When I began writing this post, there was no packaging logic for this program; it’s a standalone Haskell file. However, this file has several dependencies outside of Haskell’s standard library, so up until now I needed some way to obtain those dependencies for development.

Stage 0: ghc.withPackages

The most low-tech way that you can hack on a Haskell program using Nix is to use nix-shell to obtain a transient development environment (this is what I had done up until now).

Specifically, you can do something like this:

$ nix-shell --packages 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'

… where pkgs.mtl and pkgs.MemoTrie indicate that I want to include the mtl and MemoTrie packages in my Haskell development environment.

Inside of that development environment I can build and run the file using ghc. For example, I can use ghc -O to build an executable to run:

[nix-shell]$ ghc -O Spire.hs
[nix-shell]$ ./Spire

… or if I don’t care about optimizations I can interpret the file using runghc:

$ runghc Spire.hs

Stage 1: IDE support

Once I’m inside a Nix shell I can begin to take advantage of integrated development environment (IDE) support.

The two most common tools Haskell developers use for rapid feedback are ghcid and haskell-language-server:

  • ghcid provides a command-line interface for fast type-checking feedback but doesn’t provide other IDE-like features

  • haskell-language-server is more of a proper IDE that you use in conjunction with some editor

I can obtain either tool by exiting from the shell and creating a new shell that includes the desired tool.

For example, if I want to use ghcid then I recreate the nix-shell using the following command:

$ nix-shell --packages ghcid 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'

… and then I can tell ghcid to continuously type-check my file using:

[nix-shell]$ ghcid Spire.hs

If I want to use haskell-language-server, then I recreate the nix-shell using this command:

$ nix-shell --packages haskell-language-server 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'

… and then I can explore the code in any editor that supports the language server protocol.

Note that if you use VSCode as your editor then you may need to install some additional plugins:

… and the next section will show how to install VSCode and those plugins using Nix.

However, once you do install those plugins then you can open the file in VSCode from within the nix-shell using:

[nix-shell]$ code Spire.hs

… and once you trust the file the IDE features will kick in.

Stage 2: Global development environment

Sometimes I like to globally install development tools that are commonly shared between projects. For example, if I use ghcid or haskell-language-server across all my projects then I don’t want to have to explicitly enumerate that tool in each project’s Nix shell.

Moreover, my tool preferences might not be shared by other developers. If I share my nix-shell with other developers for a project then I probably don’t want to add editors/IDEs or other command-line tools to that environment because then they have to download those tools regardless of whether they plan to use them.

However, I don’t want to globally install development tools like this:

$ nix-env --install --file '<nixpkgs>' --attr ghcid
$ nix-env --install --file '<nixpkgs>' --attr haskell-language-server

Part of the reason I use Nix is to avoid imperatively managing my development environment. Fortunately, though, nix-env supports a more declarative way of managing dependencies.

What you can do instead is save a file like this to ~/default.nix:

let
  # For VSCode
  config = { allowUnfree = true; };

  overlay = pkgsNew: pkgsOld: {
    # Here's an example of how to use Nix to install VSCode with plugins managed
    # by Nix, too
    vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
      vscodeExtensions = [
        pkgsNew.vscode-extensions.haskell.haskell
        pkgsNew.vscode-extensions.justusadam.language-haskell
      ]; 
    };
  };

  pkgs = import <nixpkgs> { inherit config; overlays = [ overlay ]; };

in      
  { inherit (pkgs)
      # I included some sample useful development tools for Haskell.  Feel free
      # to customize.
      cabal-install
      ghcid
      haskell-language-server
      stylish-haskell
      vscode-with-extensions 
    ; 
  }     

… and once you create that file you have two options.

The first option is that you can set your global development environment to match the file by running:

$ nix-env --remove-all --install --file ~/default.nix

NOTE: At the time of this writing you may also need to add --system x86_64-darwin if you are trying out these examples on an M1 Macbook. For more details, see:

Carefully note the --remove-all, which resets your development environment to match the file, so that nothing from your old development environment is accidentally carried over into your new development environment. This makes our use of the nix-env command truly declarative.

The second option is that you can change the file to create a valid shell, like this:

let
  config = { allowUnfree = true; };

  overlay = pkgsNew: pkgsOld: {
    vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
      vscodeExtensions = [
        pkgsNew.vscode-extensions.haskell.haskell
        pkgsNew.vscode-extensions.justusadam.language-haskell
      ];
    };
  };

  pkgs = import <nixpkgs> { inherit config; overlays = [ overlay ]; };

in
  pkgs.mkShell {
    packages = [
      pkgs.ghcid
      pkgs.haskell-language-server
      pkgs.stylish-haskell
      pkgs.vscode-with-extensions
      pkgs.cabal-install
    ];
  }

… and then run:

$ nix-shell ~/default.nix

Or, even better, you can rename the file to ~/shell.nix and then if you’re already in your home directory (e.g. you just logged into your system), then you can run:

$ nix-shell

… which will select ~/shell.nix by default. This lets you get a completely transient development environment so that you never have to install anything development tools globally.

These nix-shell commands stack, so you can first run nix-shell to obtain your global development environment and then use nix-shell a second time to obtain project-specific dependencies.

My personal preference is to use the declarative nix-env trick for installing global development tools. In my opinion it’s just as elegant as nix-shell and slightly less hassle.

Stage 3: Cabal

Anyway, enough about global development tools. Back to our Haskell project!

So ghc.withPackages is a great way to just start hacking on a standalone Haskell program when you don’t want to worry about packaging up the program. However, at some point you might want to share the program with the others or do a proper job of packaging if you’re trying to productionize the code.

That brings us to the next step, which is packaging our Haskell program with a Cabal file (a Haskell package manifest). We’ll need the cabal-install command-line tool before we proceed further, so you’ll want to add that tool to your global development environment (see the previous section).

To create our .cabal file we can run the following command from the top-level directory of our Haskell project:

$ cabal init --interactive
Should I generate a simple project with sensible defaults? [default: y] n

… and follow the prompts to create a starting point for our .cabal file.

After completing those choices and trimming down the .cabal file (to keep the example simple), I get a file that looks like this:

cabal-version:      2.4
name:               spire
version:            1.0.0
license:            BSD-3-Clause
license-file:       LICENSE

executable spire
    main-is:          Spire.hs
    build-depends:    base ^>=4.14.3.0
    default-language: Haskell2010

The only thing I’m going change for now is to add dependencies to the build-depends section and increase the upper bound on base::

cabal-version:      2.4
name:               spire
version:            1.0.0
license:            BSD-3-Clause
license-file:       LICENSE

executable spire
    main-is:          Spire.hs
    build-depends:    base >=4.14.3.0 && < 5
                    , MemoTrie
                    , containers
                    , mtl
                    , pretty-show
                    , transformers
    default-language: Haskell2010

Stage 4: cabal2nix --shell

Adding a .cabal file suffices to share our Haskell package with other Haskell developers if they’re not using Nix. However, if we want to Nix-enable package our package then we have a few options.

The simplest option is to run the following command from the top-level of the Haskell project:

$ cabal2nix --shell . > shell.nix

That will create something similar to the following shell.nix file:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "default", doBenchmark ? false }:

let

  inherit (nixpkgs) pkgs;

  f = { mkDerivation, base, containers, lib, MemoTrie, mtl
      , pretty-show, transformers
      }:
      mkDerivation {
        pname = "spire";
        version = "1.0.0";
        src = ./.;
        isLibrary = false;
        isExecutable = true;
        executableHaskellDepends = [
          base containers MemoTrie mtl pretty-show transformers
        ];
        license = lib.licenses.bsd3;
      };

  haskellPackages = if compiler == "default"
                       then pkgs.haskellPackages
                       else pkgs.haskell.packages.${compiler};

  variant = if doBenchmark then pkgs.haskell.lib.doBenchmark else pkgs.lib.id;

  drv = variant (haskellPackages.callPackage f {});

in

  if pkgs.lib.inNixShell then drv.env else drv

… and if you run nix-shell within the same directory the shell environment will have the Haskell dependencies you need to build and run project using cabal:

$ nix-shell
[nix-shell]$ cabal run

… and tools like ghcid and haskell-language-server will also work within this shell, too. The only difference is that ghcid now takes no arguments, since it will auto-detect the cabal project in the current directory:

[nix-shell]$ ghcid

Note that this nix-shell will NOT include cabal by default. You will need to globally install cabal (see the prior section on “Global development environment”).

This cabal2nix --shell workflow is sufficiently lightweight that you can Nixify other people’s projects on the fly when hacking on them locally. A common thing I do if I need to make a change to a person’s project is to clone their repository, run:

$ cabal2nix --shell . > shell.nix
$ nix-shell

… and start hacking away. I don’t even need to upstream the shell.nix file I created in this way; I just keep it around locally for my own hacking.

In fact, I typically don’t want to upstream such a shell.nix file (even if the upstream author were receptive to Nix), because there are more robust Nix expressions we can upstream instead.

Stage 5: Custom shell.nix file

One disadvantage of cabal2nix --shell is that you have to re-run the command any time your dependencies change. However, if you’re willing to hand-write your own shell.nix file then you can create something more stable:

let
  overlay = pkgsNew: pkgsOld: {
    haskellPackages = pkgsOld.haskellPackages.override (old: {
      overrides = pkgsNew.haskell.lib.packageSourceOverrides {
        spire = ./.;
      };
    });
  };

  pkgs = import <nixpkgs> { overlays = [ overlay ]; };

in
  pkgs.haskellPackages.spire.env

The packageSourceOverrides is the key bit. Under the hood, that essentially runs cabal2nix for you any time your project changes and then generates your development environment from the result. You can also use packageSourceOverrides to specify non-default versions of dependencies, too:

let
  overlay = pkgsNew: pkgsOld: {
    haskellPackages = pkgsOld.haskellPackages.override (old: {
      overrides = pkgsNew.haskell.lib.packageSourceOverrides {
        spire = ./.;

        # Example of how to pin a dependency to a non-defaul version
        pretty-show = "1.9.5";
      };
    });
  };

  pkgs = import <nixpkgs> { overlays = [ overlay ]; };

in
  pkgs.haskellPackages.spire.env

… although that will only work for packages that have been released prior to the version of Nixpkgs that you’re depending on.

If you want something a bit more robust, you can do something like this:

let
  overlay = pkgsNew: pkgsOld: {
    haskellPackages = pkgsOld.haskellPackages.override (old: {
      overrides =
        pkgsNew.lib.fold
          pkgsNew.lib.composeExtensions
          (old.overrides or (_: _: { }))
          [ (pkgsNew.haskell.lib.packageSourceOverrides {
              spire = ./.;
            })
            (pkgsNew.haskell.lib.packagesFromDirectory {
              directory = ./packages;
            })
          ];
    });
  };

  pkgs = import <nixpkgs> { overlays = [ overlay ]; };

in
  pkgs.haskellPackages.spire.env

… and then you have the option to also depend on any dependency that cabal2nix knows how to generate:

$ mkdir packages

$ # Add the following file to version control to preserve the directory
$ touch packages/.gitkeep

$ cabal update

$ cabal2nix cabal://${PACKAGE_NAME}-${VERSION} > ./packages/${PACKAGE_NAME}.nix

… and that works even on bleeding-edge Haskell packages that Nixpkgs hasn’t picked up, yet.

Stage 6: Pinning Nixpkgs

All of the prior examples are “impure”, meaning that they depend on the ambient nixpkgs channel installed on the developer’s system. This nixpkgs channel might vary from system to system, meaning that each system might have different versions of nixpkgs installed, and then you run into issues reproducing each other’s builds.

For example, if you have a newer version of nixpkgs installed your Nix build for the above Haskell project might succeed, but then another developer might attempt to build your project with an older version of nixpkgs, which might select an older incompatible version of one of your Haskell dependencies.

Or, vice versa, the examples in this blog post might succeed at the time of this writing for the current version of nixpkgs but then as time goes on the examples might begin to fail for future versions of nixpkgs.

You can fix that by pinning Nixpkgs, which this post covers:

For example, we could pin nixpkgs for our global ~/default.nix like this:

let
  nixpkgs = builtins.fetchTarball {
    url    = "https://github.com/NixOS/nixpkgs/archive/0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08.tar.gz";
    sha256 = "14ann7vz7qgfrw39ji1s19n1p0likyf2ag8h7rh8iwp3iv5lmprl";
  };

  config = { allowUnfree = true; };

  overlay = pkgsNew: pkgsOld: {
    vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
      vscodeExtensions = [
        pkgsNew.vscode-extensions.haskell.haskell
        pkgsNew.vscode-extensions.justusadam.language-haskell
      ];
    };
  };

  pkgs = import nixpkgs { inherit config; overlays = [ overlay ]; };

in
  { inherit (pkgs)
      cabal-install
      ghcid
      haskell-language-server
      stylish-haskell
      vscode-with-extensions
    ;
  }

… which pins us to the tip of the release-22.05 branch at the time of this writing.

We can likewise pin nixpkgs for our project-local shell.nix like this:

let
  nixpkgs = builtins.fetchTarball {
    url    = "https://github.com/NixOS/nixpkgs/archive/0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08.tar.gz";
    sha256 = "14ann7vz7qgfrw39ji1s19n1p0likyf2ag8h7rh8iwp3iv5lmprl";
  };

  overlay = pkgsNew: pkgsOld: {
    haskellPackages = pkgsOld.haskellPackages.override (old: {
      overrides = pkgsNew.haskell.lib.packageSourceOverrides {
        spire = ./.;
      };
    });
  };

  pkgs = import nixpkgs { overlays = [ overlay ]; };

in
  pkgs.haskellPackages.spire.env

Flakes

The final improvement we can make is the most important one of all: we can convert our project into a Nix flake:

There are two main motivations for flake-enabling our project:

  • To simplify managing inputs that we need to lock (e.g. nixpkgs)
  • To speed up our shell

To flake-enable our project, we’ll save the following code to flake.nix:

{ inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/release-22.05;

    utils.url = github:numtide/flake-utils;
  };

  outputs = { nixpkgs, utils, ... }:
    utils.lib.eachDefaultSystem (system:
      let
        config = { };

        overlay = pkgsNew: pkgsOld: {
          spire =
            pkgsNew.haskell.lib.justStaticExecutables
              pkgsNew.haskellPackages.spire;

          haskellPackages = pkgsOld.haskellPackages.override (old: {
            overrides = pkgsNew.haskell.lib.packageSourceOverrides {
              spire = ./.;
            };
          });
        };

        pkgs =
          import nixpkgs { inherit config system; overlays = [ overlay ]; };

      in
        rec {
          packages.default = pkgs.haskellPackages.spire;

          apps.default = {
            type = "app";

            program = "${pkgs.spire}/bin/spire";
          };

          devShells.default = pkgs.haskellPackages.spire.env;
        }
    );
}

… and then we can delete our old shell.nix because we don’t need it anymore.

Now we can obtain a development environment by running:

$ nix develop

… and the above flake also makes it possible to easily build and run the program, too:

$ nix run    # Run the program
$ nix build  # Build the project

In fact, you can even run a flake without having to clone a repository. For example, you can run the example code from this blog post by typing:

$ nix run github:Gabriella439/spire

Moreover, we no longer have to take care of managing hashes for, say, Nixpkgs. The flake machinery takes care of that automatically for you and generates a flake.lock file which you can then add to version control. For example, the lock file I got was:

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1661617163,
        "narHash": "sha256-NN9Ky47j8ohgPhA9JZyfkYIbbAo6RJkGz+7h8/exVpE=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
        "ref": "release-22.05",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs",
        "utils": "utils"
      }
    },
    "utils": {
      "locked": {
        "lastModified": 1659877975,
        "narHash": "sha256-zllb8aq3YO3h8B/U0/J1WBgAL8EX5yWf5pMj3G0NAmc=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "c0e246b9b83f637f4681389ecabcb2681b4f3af0",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    }
  },
  "root": "root",
  "version": 7
}

… and you can easily upgrade to, say, a newer revision of Nixpkgs if you need to.

Additionally, all of the Nix commands are now faster. Specifically, the first time you run a command Nix still needs to download and/or build dependencies, but subsequent runs are faster because Nix can skip the instantiation phase. For more details, see:

Conclusion

Flakes are our final destination, so that’s as far as this post will go. There are technically some more ways that we can overengineer things, but in my experience the idioms highlighted in this post are the ones that provide the highest power-to-weight ratio.

The key thing to take away is that the Nixpkgs Haskell infrastructure lets you smoothly transition from simpler approaches to more powerful approaches, and even the final flake-enabled approach is actually not that complicated.

4 comments:

  1. Great stuff. The exposition assumes that cabal and cabal2nix are already in the environment. It seems it would be useful to show how to "install" them with nix too.

    ReplyDelete
    Replies
    1. Rereading, I see you said to add cabal-install to the global env --- so that takes care of that! False alarm.

      Delete
  2. There's an intermediate stage between step 4 (calling cabal2nix manually) and 5 (a full-fledged hand-rolled nix module), which is a one-line nix module with `callCabal2nix`. Along the lines described here: https://bytes.zone/posts/callcabal2nix/ (only they suggest to duplicate the call to callCabal2nix in shell.nix, and I prefer to simply import default.nix with that call into shell.nix). So far, it seems best benefit/cost solution to me if I want to work mainly with the .cabal file but build with Nix when available.

    A more comprehensive solution worth mentioning: https://github.com/utdemir/hs-nix-template/

    Question. I have a constant trouble using pure cabal on NixOS: whenever I try to build with cabal anything non-trivial, I fail because zlib.dev is not available. It can be fixed by entering a nix-shell -p zlib.dev, of course, but then the build starts from scratch and that's annoying. I tried adding zlib.dev into systemPackages but that doesn't help. I wonder if you know to solve it.

    ReplyDelete
  3. This is a good introduction! Note that there is also https://github.com/srid/haskell-flake which abstracts all of this as a flake module. There is also https://github.com/srid/haskell-template which uses this.

    ReplyDelete