Monday, May 21, 2018

How I evaluate Haskell packages

This post summarizes the rough heuristics that I use for evaluating Haskell packages. Usually I use these rules of thumb when:

  • choosing between multiple similar packages
  • deciding whether to depend on a package at all

Some of these guidelines work for other programming languages, too, but some of them are unique to Haskell. Even if you are a veteran programmer in another language you still might find some useful tidbits here when approaching the Haskell ecosystem for the first time.

I've organized the metrics I use into five categories:

  • "Large positive" - Things that impress me
  • "Small positive" - Positive features that make me more likely to adopt
  • "Irrelevant" - Things I never pay attention to when judging package quality
  • "Small negative" - Things that turn me off
  • "Large negative" - Huge red flags

Large positive

  • High quality documentation

    Excellent documentation is one of the strongest signals of maintainer commitment to a package. Extensive documentation is very expensive to keep up to date with a package because documentation is difficult to machine check in general.

    You can definitely have the build check that type signatures or doctests are correct, but anything more elaborate usually requires human intervention to keep up to date.

    Disclaimer: I'm biased here because I'm a huge proponent of documenting things well.

  • Impossible to use incorrectly

    One of the nice features of Haskell is that you can use the type system to make impossible states unrepresentable or enforce invariants to guarantee proper use. I strongly prefer libraries that take advantage of this to rule out errors at compile time so that I can sleep soundly at night knowing that my work built on top of a solid foundation.

  • On Stackage

    All packages on Stackage are guaranteed to build and have a known good build plan (called a Stackage "resolver") that both stack and cabal can reuse. All packages on Stackage "play nice" with each other, which greatly reduces the total cost of ownership for depending on these Haskell packages.

    Being on Stackage is also a good signal that the package is maintained by somebody because packages require some upkeep to ensure that they continue to build against the latest version of other packages.

  • Competitive Benchmarks

    I strongly prefer packages that post benchmarks comparing their package to their competitors. Such a comparison implies that the author went to the trouble of using and understanding alternative packages and using them idiomatically. That perspective often improves the quality and design of their own package.

    Also, competitive benchmarks are rare and require significant upkeep, so they double as strong signal of maintainer commitment.

  • 100+ reverse dependencies

    You can check the packages that depend on a given package (a.k.a. "reverse dependencies") using the following URL:

    https://packdeps.haskellers.com/reverse/${package-name}

    Packages that have a 100 or more reverse dependencies have substantial buy-in from the Haskell ecosystem and are safe to depend on. You're also likely to depend on these packages anyway through your transitive dependencies.

    (Suggested by: Tim Humphries)

Small positive

  • One complete example

    This means that the package has one "end-to-end" example showing how all the pieces fit together. I prefer packages that do this because they don't force me to waste a large amount of time reinventing what the author could have communicated in a much shorter amount of time.

    You would be surprised how many Haskell packages have extensive Haddock coverage but still lack that one overarching example showing how to use the package.

  • 100+ downloads in last 30 days

    Download statistics are not a very strong signal of activity on Hackage since most Haskell build tools (i.e. cabal, stack, Nix) will cache dependencies. These tools also don't require frequent cache invalidations like some other build tools which shall not be named.

    However, having 100+ downloads in the last 30 days is a reasonable signal that enough people were recently able to get the package working. That means that if anything goes wrong then I assume that there must be some solution or workaround.

  • Maintained/used/endorsed by a company

    I think this is a pretty obvious signal. Similar to download metrics, this implies that somebody was able to extract something commercially valuable from this package.

  • Pure or mostly pure API

    Packages that provide pure APIs are easier to test and easier to integrate into a code base. This reduces the total cost of ownership for that dependency.

  • Upper bounds on dependencies that are up to date

    I'm not going to take a strong stance on the "great PVP war" in this post. All I will note is that if a package has upper bounds that are up to date then that is a good sign that the package is actively maintained (whether or not you agree that upper bounds are a good idea).

    In other words, upper bounds are one way that a package author can "virtue signal" that they have the time and resources to maintain a package.

Irrelevant

  • On Hackage

    Many people new to Haskell ecosystem assume that there is some sort of quality bar to be on Hackage just because Hackage is not GitHub. There is no such quality bar; anybody can obtain a Hackage account and upload their packages (even joke packages).

  • Stability field

    The stability field of packages fell out of fashion years ago. Also, in my experience people are poor judges of whether or not their own packages will be stable. I prefer to look at the frequency of releases and how often they are breaking changes as empirical evidence of a package's stability.

  • Version number

    First off, make sure you understand PVP. The key bit is that the first two components of a version number represent the major version of a package. That means that if the package goes from version 1.0 to 1.1 then that signals just as much of a breaking change to the API as if the package went from version 1.0 to 2.0. The usual rule of thumb for interpreting version numbers is:

    X.Y.Z.A
    ↑ ↑ ↑ ↑
    | | | |
    | | | Non-breaking change that is invisible (i.e. documentation or refactor)
    | | | Note: Not all packages use a fourth number in their versions
    | | |
    | | Non-breaking change
    | |
    | Breaking change without fanfare
    |
    Breaking change with great fanfare

    The absolute version number basically does not matter. There are many widely used and mature packages on Hackage that are still version 0.*. This can be jarring coming from another language ecosystem where people expect a mature package to be at least version 1.0.

  • Votes

    Hackage 2 supports voting on packages but I've never consulted a package's votes when deciding whether or not to adopt. I prefer quality signals that cannot be easily faked or gamed.

  • Tests

    I've never checked whether or not I'm using a well-tested package or not. Maybe that's not a good idea, but I'm being honest about how I evaluate packages.

    This probably comes as a surprise to a person coming to Haskell from another language ecosystem, especially a dynamically typed language that rely heavily on tests to ensure quality. Haskell programmers lean much more on the type system to enforce and audit quality. For example, type signatures show me at a glance if a package is overusing side effects or deals with too many states or corner cases that I need to account for.

Small negative

  • Upper bounds that are not up to date

    The other nice thing about adding upper bounds to a package is that they also accurately signal when a package is no longer maintained. They require constant "gardening" to keep up to date so when an author abandons a package they naturally transform from a positive signal into a negative signal simply by virtue of inaction.

  • Large dependency footprint

    This used to be a much bigger issue before the advent of Stackage. People would aggressively trim their dependency trees out of fear that one bad dependency would cause them to waste hours finding a valid build plan.

    I still spend some time these days trying to minimize my dependency footprint, mainly so that I don't need to respond as often to breaking changes in my immediate dependencies. However, this is a relative minor concern in comparison to how bad things used to be.

  • Uses linked lists inappropriately

    The #1 source of Haskell performance degradation is keeping too many objects on the heap and linked lists trash your heap if you materialize the entire list into memory (as opposed to streaming over the list so that you only materialize at most one element at any time).

    I try to avoid dependencies that use linked lists unless they are the optimal data structure because I will be paying the price for those dependencies every time my program runs a garbage collection.

  • Frequent breaking API changes

    Breaking API changes are not a really big deal in Haskell due to the strong type system. Breaking API changes are mainly a minor maintenance burden: if you depend on a library that releases breaking changes all the time you usually only need to spend a few minutes updating your package to build against the latest API.

    This matters more if you maintain a large number of packages or if you have a huge number of dependencies, but if you don't then this is a relatively small concern.

  • Idiosyncratic behavior

    "Idiosyncratic" means that the package author does something weird unique to them. Some examples include using a private utility library or custom Prelude that nobody else uses or naming all of their classes C and all their types T (you know who you are).

    When I see this I assume that the package author has no interest in building consensus or being a "well-behaved" member of the Haskell ecosystem. That sort of package is less likely to respond or adapt to user feedback.

Large negative

  • Uses String inappropriately

    This is a special case of "Uses linked lists inappropriately" since the default String type in Haskell is stored as a linked list of characters.

    Strings tend to be rather large linked lists of characters that are commonly fully materialized. This implies that even modest use of them will add lots of objects to the heap and tax the garbage collector.

  • No documentation

    If I see no documentation on a package I "nope" right out of there. I assume that if a maintainer does not have time to document the package then they do not have time to address user feedback.

  • Not on Hackage

    I assume that if a package is not on Hackage that the maintainer has no interest in supporting or maintaining the package. The bar for adding a package to Hackage is so low that the absence of a package on Hackage is a red flag.

    Note that this does not imply that packages on Hackage will be supported. There are packages on Hackage where the author does not intend to support the package.

    Note: a reader pointed out that this is confusing given that I treat "on Hackage" as "Irrelevant". Perhaps another way to phrase this is that being on Hackage is a necessary but not sufficient signal of maintainer support.

  • Maintainer ignoring pull requests

    This is obviously the clearest sign that a maintainer does not respond to user feedback. There are fewer things that upset me more than a maintainer that completely ignores volunteers who submit pull requests to fix bugs.

Conclusion

Hopefully that gives you some quick and dirty heuristics you can use when evaluating Haskell packages for the first time. If you think that I missed something important just let me know and I'll update this post.

4 comments:

  1. A few comments:

    Since you mention pull requests: is being on github a criterion?

    I think the 100+ reverse dependencies are rather hard to reach for more specialized packages.

    Depending on what you consider "inappropriate", I'd disagree with the String criterion. In places where performance does not matter they are often more readable than Text and save a dependency. Also (I admit I'm nitpicking) I don't read the Haskell report as requiring Strings to be stored as a linked list of characters. They just have to behave as if they were. I know that's hard in the presence of lazyness, but certainly not impossible. And I hope this to change if we ever get more than one relevant compiler.

    ReplyDelete
    Replies
    1. Being on GitHub is not a criterion as long as the author provides some way to accept patches. For example, GitLab or BitBucket would probably be fine, too.

      Packages don't have to have 100+ reverse dependencies for me to adopt them. "Strong positive" does not imply a necessary condition for me to adopt a package

      I also believe that the Haskell report does require `String` to be a linked list of characters. The report specifies a standard implementation of the Prelude:

      https://www.haskell.org/onlinereport/standard-prelude.html

      ... which defines:

      type String = [Char]

      Delete
  2. If not being on hackage is a large negative, then being on hackage is surely not irrelevant?

    ReplyDelete
    Replies
    1. Yeah, somebody else pointed out the same inconsistency. I added a clarifying comment:

      > Note: a reader pointed out that this is confusing given that I treat "on Hackage" as "Irrelevant". Perhaps another way to phrase this is that being on Hackage is a necessary but not sufficient signal of maintainer support.

      Delete