Sunday, June 16, 2019

The CAP theorem for software engineering

cap

The CAP theorem says that distributed computing systems cannot simultaneously guarantee all three of:

  • Consistency - Every read receives the most recent write or an error

  • Availability - Every request receives a (non-error) response - without the guarantee that it contains the most recent write

  • Partition tolerance - The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes

Source: CAP theorem - Wikipedia

Since we cannot guarantee all three, we must typically sacrifice at least one of those guarantees (i.e. sacrifice availability or sacrifice partition tolerance).

However, what if we were to squint and apply the CAP theorem to another distributed system: a team of software engineers working towards a common goal.

In particular:

  • What if our data store were a distributed version control system?

  • What if our “nodes” were software developers instead of machines?

If we view engineering through this lens, we can recognize many common software engineering tradeoffs as special cases of CAP theorem tradeoffs. In other words, many architectural patterns also require us sacrifice at least one of consistency, availability, or partition tolerance among developers.

Before we get into examples, I’d like to review two points that come up in most discussions of the CAP theorem:

Partition tolerance

What does it mean to sacrifice partition tolerance? In the context of machines, this would require us to rule out the possibility of any sort of network failure.

Now replace machines with developers. We’d have to assume that people never miscommunicate, lose internet access, or fetch the wrong branch.

The possibility of partitions are what make a system a distributed system, so we’re usually not interested in the option of sacrificing partition tolerance. That would be like a computing system with only one machine or a team with only one developer.

Instead, we’ll typically focus on sacrificing either availability or consistency.

Spectrums of tradeoffs

In most systems you’re always sacrificing all three of consistency, availability, and partition tolerance to some degree if you look closely enough. For example, even a healthy machine is not 100% available if you consider that even the fastest network request still has an irreducible delay of around a few hundred microseconds on today’s machines.

In practice, we ignore these vanishingly small inconsistencies or inavailabilities, but they still illustrate a general pattern: we can think of system health/availability/consistency as spectrums rather than boolean options.

For example, if we say we choose availability over consistency, we really mean that we choose to make our system’s unavailability vanishingly small and that our system could be consistent, but not all the time. Indeed, if our hardware or network were both fast and extremely reliable we could enjoy both high consistency and high availability, but when things fail then we need to prioritize which of consistency or availability that we sacrifice to accommodate that failure.

We can also choose to sacrifice a non-trivial amount of both availability and consistency. Sometimes exclusively prioritizing one or the other is not the right engineering choice!

With those caveats out of the way, let’s view some common software engineering tradeoffs through the lens of the CAP theorem.

Monorepo vs. Polyrepo

In revision control systems, a monorepo (syllabic abbreviation of monolithic repository) is a software development strategy where code for many projects are stored in the same repository

A “polyrepo” is the opposite software development strategy where each project gets a different source repository. In a monorepo, projects depend on each other by their relative paths. In a polyrepo, a project can depend on another project by referencing a specific release/revision/build of the dependency.

The tradeoff between a monorepo and a polyrepo is a tradeoff between consistency and availability. A monorepo prioritizes consistency over availability. Conversely, a polyrepo prioritizes availability over consistency.

To see why, let’s pretend that project A depends on project B and we wish to make a breaking change to project B that requires matching fixes to project A. Let’s also assume that we have some sort of continuous integration that ensures that the master branch of any repository must build and pass tests.

In a polyrepo, we can make the breaking change to the master branch of project B before we are prepared to make the matching fix to project A. The continuous integration that we run for project B’s repository does not check that other “downstream” projects that depend on B will continue to build if they incorporate the change. In this scenario, we’ve deferred the work of integrating the two projects together and left the system in a state where the master branch of project B is not compatible with the master branch of project A. (Note: the master branch of project A may still build and pass tests, but only because it depends on an older version of project B).

In a monorepo, we must bundle the breaking change to project B and the fix to project A in a single logical commit to the master branch of our monorepo. The continuous integration for the monorepo prevents us from leaving the master branch in a state where some of the projects don’t build. Fixing these package incompatibilities up-front will delay merging work into the master branch (i.e. sacrificing availability of our work product) but imposing this restriction ensures that the entire software engineering organization has a unified view of the codebase (i.e. preserving consistency).

To make the analogy precise, let’s revisit the original definitions of consistency, availability, and partition tolerance:

  • Consistency - Every read receives the most recent write or an error

  • Availability - Every request receives a (non-error) response - without the guarantee that it contains the most recent write

  • Partition tolerance - The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes

… and change them to reflect the metaphor of a distributed team of developers collaborating via GitHub/GitLab:

  • Consistency - Every git pull receives the latest versions of all dependencies

  • Availability - Every pull request succeeds - without the guarantee that it contains the latest versions of all dependencies

  • Partition tolerance - git operations continue to work even if GitHub/GitLab is unavailable

Trunk based development vs. Long-lived branches

Our previous scenario assumed that each repository was using “trunk-based development”, defined as:

A source-control branching model, where developers collaborate on code in a single branch called “trunk” [and] resist any pressure to create other long-lived development branches by employing documented techniques.

Source: trunkbaseddevelopment.com

The opposite of trunk-based development is “long-lived branches” that are not the master branch (i.e. the “trunk” branch).

Here are some examples of long-lived branches you’ll commonly find in the wild:

  • A develop branch that is used as a the base branch of pull requests. This develop branch is periodically merged into master (typically at release boundaries)

  • Release branches that are suported for months or years (i.e. long-term support releases)

  • Feature branches that people work on for an extended period of time before merging their work into master

The choice between trunk-based development and long-lived branches is a choice between consistency and availability. Trunk-based development prioritizes consistency over availability. Long-lived branches prioritize availability over consistency.

To see why, imagine merging a feature branch over a year old back into the master branch. You’ll likely run into a large number of merge conflicts because up until now you sacrificed consistency by basing your work on an old version of the master branch. However, perhaps you would have slowed down your iteration speed (i.e. sacrificing availability of your local work product) if you had to ensure that each of your commits built against the latest master.

You might notice that trunk-based development vs. long-lived branches closely parallels monorepo vs. polyrepo. Indeed, organizations that prefer monorepos also tend to prefer trunk-based development because they both reflect the same preference for developers sharing a unified view of the codebase. Vice versa, organizations that prefer polyrepo also tend to prefer long-lived branches because both choices emerge from the same preference to prioritize availability of developers’ work product. These are not perfect correlations, though.

Continuous integration vs. Test team

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

Source: thoughtworks.com - Continous Integration

So far we’ve been assuming the use of continuous integration to ensure that master stays “green”, but not all organization operate that way. Some don’t use continuous integration and rely on a test team to identify integration issues.

You can probably guess where this is going:

  • Continuous integration prioritizes consistency over availability
  • Use of a test team prioritizes availability over consistency

The more you rely on continuous integration, the more you need to catch (and fix) errors up front since the error-detection process is automated. The more you rely on a test team the more developers tend to defer detection of errors and bugs, leaving the system in a potentially buggy state for features not covered by automated states.

Organizations that use a test team prioritize availability of developers’ work product, but at the expense of possibly deferring consistency between components system-wide. Vice-versa, organizations that rely on continuous integration prioritize consistency of the fully integrated system, albeit sometimes at the expense of the progress of certain components.

Spectrums

Remember that each one of these choices is really a spectrum. For example:

  • Many monorepos are partially polyrepos, too, if you count their third-party dependencies. The only true monorepo is one with no external dependencies

  • The distinction between trunk-based development and long-lived branches is a matter of degree. There isn’t a bright line that separates a short-lived branch from a long-lived one.

  • Many organizations use a mix of continuous integration (to catch low-level issues) and a test team (to catch high-level issues). Also, every organization has an implicit test team: their customers, who will report bugs that even the best automation will miss.

Conclusion

This post is not an exhaustive list of software engineering tradeoffs that mirror the CAP theorem. I’ll wager that as you read this several other examples came to mind. Once you recognize the pattern you will begin to see this tension between consistency and availability everywhere (even outside of software engineering).

Hopefully this post can help provide a consistent language for talking about these choices so that people can frame these discussions in terms of their organization’s core preference for consistency vs availability. For example, maybe in the course of reading this you noticed that your organization prefers availability in some cases but consistency in others. Maybe that’s a mistake you need to correct or maybe it’s an inevitability since we can never truly have 100% availability or 100% consistency.

You might be interested in what happens if you take availability or consistency to their logical conclusion. For example, Kent Beck experiments with an extreme preference for consistency over availability in test && commit || revert. Or to put it more humorously:

On the other hand, if you prioritize availability over consistency at all costs you get … the open source ecosystem.

This is not the first post exploring the relationship between the CAP theorem and software development. For example, Jessica Kerr already explored this idea of treating teams as distributed systems in Tradeoffs in Coordination Among Teams.

Tuesday, May 14, 2019

Release early and often

release-frequently

This post summarizes the virtues of cutting frequent releases for software projects. You might find this post useful if you are trying to convince others to release more frequently (such as your company or an open source project you contribute to).

Easing migration

Frequent releases provide a smoother migration path for end-users of your software.

For example, suppose that your software is currently version “1.0” and you have two breaking changes (“A” and “B”) that you plan to make. Now consider the following two release strategies

  • Release strategy #0 - More frequent releases

    * Version 1.0
        * Initial release
    
    * Version 2.0
        * BREAKING CHANGE: A
    
    * Version 3.0
        * BREAKING CHANGE: B
  • Release strategy #1 - Less frequent releases

    * Version 1.0
        * Initial release
    
    * Version 2.0
        * BREAKING CHANGE: A
        * BREAKING CHANGE: B

The first release strategy is better from the end-user’s point of view because they have the option to upgrade in two smaller steps. In other words, they can upgrade from version 1.0 to version 2.0 and then upgrade from version 2.0 to version 3.0 at a later date.

Under both release strategies users can elect to skip straight to the latest release if they are willing to pay down the full upgrade cost up front, but releasing more frequently provides users the option to pay down the upgrade cost in smaller installments. To make an analogy: walking up a staircase is easier than scaling a sheer cliff of the same height.

In particular, you want to avoid the catastrophic scenario where a large number of users refuse to upgrade if one release bundles too many breaking changes. The textbook example of this is the Python 2 to 3 upgrade where a large fraction of the community refused to upgrade because too many breaking changes were bundled into a single release instead of spread out over several releases.

Keeping trains running on time

You don’t need to delay a release to wait for a particular feature if you release frequently. Just postpone the change for the next release if it’s not ready. After all, if you release frequently then the next release is right around the corner.

Conversely, if you release infrequently, you will frequently run into the following vicious cycle:

  • Important feature X is close to completion but perhaps not quite ready to merge

    Perhaps the feature has insufficient tests or there are unresolved concerns during code review

  • A new release is about to be cut

    Should you wait to merge feature X? It might be a long time (3 months?) before the next release, even though X could be ready with just 1 more week of work.

  • You choose to delay the release to wait for important feature X

  • Now another important feature Y requests to also slip in before the release

    … further delaying the release

  • 3 months have passed and you still haven’t cut the release

    New features keep (justifiably) slipping in out of fear that they will have to otherwise wait for the next release

Eventually you do cut a release, but each iteration of this process decreases the release frequency and compounds the problem. The less frequently you release software the more incentive to slip in last-minute changes before the release cutoff, further delaying the release. Even worse, the longer you wait to cut each release the greater the pressure to compromise on quality to get the release out the door.

Sticking to a strict and frequent release schedule staves off this vicious cycle because then you can always safely postpone incomplete features to the next release.

Avoiding “crunch time”

Infrequent “big bang” releases create pressure for developers to work excessive hours in the lead up to a release cutoff. This can happen even when developers are unpaid, such as on open source projects: the peer pressure of holding up the release for others can induce people to work unhealthy schedules they wouldn’t work otherwise.

I won’t claim that frequent release schedules will prevent paid developers from working late nights and weekends, but at least management can’t hide behind a looming release deadline to justify the overtime.

Accelerating the feedback loop

Releases are opportunities to correct course because you don’t know how users will react to a feature until you put the feature into their hands. If you implement a feature and the next release is 3 month away, that’s 3 months where you don’t know if the feature is what the user actually needs.

Even worse: suppose that the first implementation of the feature does not do what the user wants: now you have to wait another 3 months to get the next iteration of the feature into their hands. That slow feedback loop is a recipe for a poorly-designed product.

Incentivizing automation

Fast release cycles force you to automate and accelerate release-related processes that you would otherwise do manually (i.e. continuous integration), including:

  • Testing
  • Publication of software artifacts
  • Collecting quality and health metrics

That automation in turn means that you spend more time in the long run developing features and less time delivering them to end users.

Conclusion

Releasing more frequently isn’t free: as the previous section suggests, you need to invest in automation to be able to make frequent releases a reality.

However, I do hope that people reading this post will recognize when symptoms of infrequent releases creep up on them so that they can get ahead of them and make the case to others to invest in improving release frequency.

Thursday, February 21, 2019

Dhall Survey Results (2019-2019)

2018-2019-dhall-survey

The results from the latest Dhall survey are in, which you can view here:

… and I would like to thank everybody who took the time to participate in the survey!

Adoption

This year 61 people completed the survey (compared to 19 last year), so we have a greater sample size to inform the future direction of the language.

Here is the breakdown of how often people used Dhall:

  • 07 (11.7%) - Never used it
  • 22 (36.7%) - Briefly tried it out
  • 11 (18.3%) - Use it for my personal projects
  • 19 (31.7%) - Use it at work
  • 01 (01.7%) - Trying to convince work people to use it

I was pleasantly surprised by the fact that more people use Dhall at work than those who use Dhall solely for personal projects. That suggests to me that people who do enjoy using Dhall do not have difficulty getting permission from their manager or coworkers to also use Dhall at work. I could be wrong, though, because there might be sampling bias or insufficient data to get accurate numbers. Those numbers also don’t necessarily imply that people have convinced their coworkers to use Dhall and I plan to update the survey next year to ask about that, too.

Let me know if you think I’m wrong and you have difficulty getting permission to use Dhall at work. Providing a smooth adoption path has always been a high priority for me because I know from experience the difficulty of introducing new tools at work.

Reasons to adopt

Most people confirmed that they use Dhall for ops-related use cases:

Kubernetes only at the moment, but more to follow

To generate Kubernetes YAML configuration files

Kubernetes, Terraform and application configs

I’m still evaluating it. Currently I’m generating Prometheus (YAML) configuration from it.

As of now I am trying it to use it for generating concourse pipelines. I work at a very ops heavy company, I can see couple of our proprietary tools could also leverage dhall.

Kubernetes, custom config for Haskell projects

configuring GoCD yaml pipelines, https://github.com/tomzo/gocd-yaml-config-plugin

Generating config for Terraform, Packer and for our application configuration.

I’m trialing it for configuring various tools configured with yaml or json like docker-compose and some internal haskell tools.

Description language for a Terraform replacement tool.

… generate yaml files then read by ansible

We use it to validate we have all the required (and no excess) variables set for our deploy scripts via type checking. If we add a variable to one environment, but miss others, the builds abort before being shipped to the world. …

Configuration files for Elm, Packer …

Replace Nix code with something saner and generate configuration for Kubernetes.

Kubernetes, Kops, Concourse, Terraform, application config

The other specific use cases I noticed were:

  • configuring build tools
  • command-line interface
  • backend service configuration
  • wire format for transmitting code

This feedback is consistent with my understanding of how Dhall is used in the wild.

Each year I try port a difficult configuration file format to Dhall to stress test the language design. Last year I ported the nethack configuration format to Dhall and this year I plan to port an ops-related configuration format with a weakly-typed schema to inform the language design process.

Document best practices

Survey respondents very commonly requested use cases, cook books, design patterns, real-world examples, and project structure guidelines. This was far-and-away the most consistent feedback from the survey:

More real world examples, like CLI application config files. Or web server-client communication examples.

More blog posts/tutorials on use cases

Resources on the next steps beyond learning syntax: how to structure Dhall code, how to organise your files, design patterns, etc.

Guides/pointers to regular things like newtypes, String equality, Homogeneous record/map constraints?

A set of documented best practices for doing common things

What would make me really happy is to see some guidelines, patterns or examples for how to support evolving schemas for clients you don’t control. …

Full list of possible uses with detailed examples and comparisons with similar tools.

Add several complete realistic examples (besides the existing snippets).

Learning curve through examples, starter apps, tutorials

… docs, examples

… basic usage patterns (newtypes, sums, comparisons)

A use-case.

I think Dhall should develop on making its way into common use cases boosting its clout in the industry.

End-user ergonomics and patterns. Pain points will come up rapidly if you start writing libraries to configure various popular tools. Those pain points should guide possible changes to the language, and the patterns developed need to be put front and center in a cookbook or something because there are many ways to tackle problems in Dhall and the best ways aren’t always obvious.

Widely used, typical use cases. Most people configure something “simple”, show me how Dhall can improve my workflow (validation against a “schema”, deduplication, sharing of common config between applications). The initial impression from the Dhall Github and website is that its very powerful but most configuration is dead simple, just very verbose…

Documentation.

Docs, …

examples how to approach a domain generically…

Last time I checked, the Dhall doc was very focused on the language itself, and the configuration part was kind of forgotten. Also, having the example with unicode syntax, while cool, make them hard/impossible to type along. I think having a few more docs about basic usecase to translate an existing json/yaml config into dhall would help adoption.

Nothing fancy with language feature, simply having a statically typed configuration, and optionally a type safe way to read it and map it to host language structure would be a very good start already.

Documentation should provide more pointers to idiomatic code. We based most of our current development off the dhall-kubernetes model which leads to dozens of boilerplate type imports at the beginning of files just to see how dhall-terraform makes all of their types available in a single expression ( https://github.com/blast-hardcheese/dhall-terraform/blob/master/Terraform/Providers/Datadog/Types.dhall ). Writing a cookbook would help with this.

This is also consistent with the highest rated funding mechanism: “Books and/or merchandise” (See below). I suspect that most people who selected that option were interested in the book.

This feedback surprised me because I was still overly focused on improving the documentation for lower-level language features. So thank you to everybody who pointed this out because this was a huge blind spot of mine.

So my plan is to initially focus on documenting the current state-of-the-art best practices as one of my highest priorities and keeping that document up to date as the language evolves. This sort of technical writing is also the kind of thing I enjoy doing anyway! :)

Language bindings and/or integrations

People also very commonly requested more language bindings and integrations with existing tools:

python library. having haskell is great, but we have a lot of python and C# as well and it would be great to use it from there too

Bindings for languages I have to work with (Java/Python).

Integration with NixOS

Tighter integration with Nix

… would like to use it for nix

Compilation to Java, Haskell, etc.

Scala integration, complete Kubernetes API in dhall-kubernetes

More DSLs a-la dhall-kubernetes. IMHO the assurance to have a well-formed (in the sense of API conformance) config, is a huge plus.

A complete Scala implementation, as we use almost only Scala at work

A JSON->dhall process would be a big help to import complex data sources. There is a dhall-terraform effort underway, but we have decently large number of terraform (HCL) files, and being able to convert HCL->json->dhall would mean I’m completely free of some really annoying restrictions HCL imposes on me.

Import from Jason/yaml

Python interpreter and syntax highlight in github

… more language/tool integration

Language Bindings …

More language bindings (e.g. Go and Python are two big Ops markets)

I think golang bindings would potentially help with lots of the things I care about in my day job: terraform, prometheus, kubernetes, cloud foundry…

other language integration

Bindings/libraries for other languages.

Language bindings, …

Getting more software to adopt dhall is a novel goal. But I see the modern ops world is full of golang software which would require golang bindings for dhall. It’s probably not a good language to write the bindings (or anything really), but it’s popularity might mean that for dhall’s success golang bindings are necessary.

Having more good language implementation, into industrial programming languages, and get rid of Yaml.

Bindings to increase adoption.

More languages…Ocaml maybe?

Cleanbindings https://wiki.clean.cs.ru.nl/

More integrations.

Using dhall from more languages would help adoption IMO. A statically-compiled lib with bindings in JS, python, java/scala would be good (and less work than implementing the language itself in all those languages). More projects like dhall-kubernetes are also a good way to drive adoption.

Being able to more quickly add it to existing code bases would be helpful. Going from JSON->dhall and then dhall being able to spit out nix/json, without needing a linter for the down stream languages makes life easier. …

I do plan to address the request to import Dhall values from JSON this year and this issue tracks the discussion and work on that:

On the other hand, I do not yet plan to personally contribute a new language binding, mainly because I want to distribute control of the language evolution process. Each reimplementation of the language gets a vote on proposed language changes and if I contribute more than one language binding then I get a disproportionate voice. Instead, I focus on making it as easy as possible for others to port the language by improving the documentation and automation surrounding the language standard.

I have no intention of becoming a benevolent dictator-for-life over the language. For example, the other voting member (Fabrizio Ferrai, who maintains the Clojure bindings to Dhall) plans to author their own interpretation of the survey feedback to parallel this post. Also, hopefully there will be two new voting members this year since the Python and PureScript language bindings are getting close to completion.

However, I can still help recruit donations for both new and existing language bindings. If you work on any language binding to Dhall and you would like to be paid for your work then set up a Patreon account or similar donation mechanism and I will help advertise that. This is an area where I recommend using distributed, non-corporate sources of funding to ensure that the language evolution process remains as democratic as possible.

Performance

Performance was another common theme:

Faster performance; …

Speed. I’d really like to mix it into things more freely as a templating language, but it’s too noticeable a slow down.

If dhall-kubernetes finally becomes fast, our whole infrastructure config set (at Wire) can move there

Performance and UX. Performance currently is very bad for large dhall projects (like dhall-kubernetes)

Performance for larger scale usage (such as nix)

Speed.

I agree with this feedback and improving performance is generally a never-ending process because every time I improve the interpreter performance people begin using the language for even more ambitious projects that strain the interpreter (myself included).

Because performance is an open-ended problem this is one of the areas I’m most likely to solicit donations to fund somebody to improve interpreter performance. That would also free me up to do more technical writing for the Dhall ecosystem.

If you are able and willing to improve the performance of the interpreter then let me know and I’ll work with you to secure some donation mechanism to fund your work. I am reasonably confident there are a few companies using Dhall that would fund improvements to interpreter performance. Similarly, if you are a company that can spare some budget to fund performance improvements, also reach out to me :)

Default record values

Another common theme was that people are struggling to port Dhall to some configuration formats that have optionally present keys:

Optionally present keys in a dictionary

… Defaults for records

Having optionals that don’t need to be maked with None when not present

I’d love Dhall to become a type-safe version of Nix (the language) or a handy language for configuration files, but there are no optionally present keys in dictionaries now, so I cannot use Dhall for config files yet. Currently I’ll just stick with YAML

Dhall actually does have a design pattern for this sort of idiom, which is to override a default record with the present values, like this:

defaultRecord // { foo = 1 }

… so this might just be another case for needing better documentation for best practices. However, if you are not satisfied with that idiom then I invite you to open an issue on the issue tracker with your thoughts on the subject:

Language stability / migration

People are beginning to request greater language stability as they begin to adopt Dhall at work:

Greater stability in the core language and prelude. There have been a bunch of changes last year, that while I agree with their purpose, I cranked down on adding more usage of Dhall until the ecosystem is more stable to avoid version brittleness

Smoother transitions between standard versions. Maybe if the last two versions of standard were supported we could provide deprecation warnings, and have something like rehash subcommand. This would allow us to update hash of existing imports only if existing (old) hash is valid.

  1. Stabilizing the language standard; 2. Stabilizing the prelude API.

… stability of upstream packages

There is one feature in the pipeline which should improve things here, which is support for stable hashes (i.e. semantic integrity checks):

This pain point was one of the most commonly cited issues with upgrading to the next language version and this will be available in the next release of the language standard and the Haskell interpreter (along with backwards-compatible support for older semantic integrity checks).

Besides that, I don’t expect new language features to be too disruptive at this point. At this point I expect most new features to the language to be additive to the language and the most disruption that users might run into is some identifier becoming a reserved keyword.

Also, the Haskell implementation takes pains to make the migration process smoother by providing dhall lint support for migrating old code to new idioms. For example, dhall lint will automatically rewrite the old syntax for Optional literals to use the newer syntax.

One of the features of the Dhall language’s standardization process is that every new language binding adds a new vote on proposed changes, which raises the bar for changing the language. So as the language grows more mainstream I expect the language standard to stabilize due to this effect.

No Unicode in examples

Three respondents disliked the use of Unicode in tutorials and examples. This was an unusually specific point of feedback that recurred a few times throughout the responses:

Get rid of unicode. It is cool but it really scares off beginners to see examples with unicode.

Better documentation, without unicode symbols.

removing unicode syntax

I don’t have any objection to switching examples to use ASCII instead of Unicode, so I will do that. I think Unicode is prettier, but if it gets in the way of the learning process then I’ll switch to ASCII.

For people who hate Unicode syntax, period, the dhall command-line tool also supports an --ascii flag for emitting ASCII when formatting or linting code.

Developer experience

Most of the developer experience feedback revolved around editor support, although there were a few other recommendations:

Better error messages (I mean, I can tell a lot of work has gone into them, which i really appreciate, but I am still often baffled by them)

A community mailing list where I could ask dumb questions when I get stuck. There’s a stackoverflow tag but it seems quite low-traffic so I was put off by that.

… editor support

… editor integrations, …

Release on OSX/Windows …

Type / syntax errors that are easier to visually parse.

Ergonomics - language server, support for auto-complete in editors, seeing type errors in the editor without needing to run the dhall executable separately

I love Gabriel’s idea on LSP support to provide support for all editors in his state of Dhall address blog post. Making the development experience (including stability of the language) will be key to letting the configs loose outside of the services team which is a little more adept at employing functional and typed methods and tools than our Ruby applications developers on the adjacent team we support.

I’d love to be able to specify a dhall file as an argument to dhall/dhall-to-* rather than feeding to STDIN. Pipes and redirection can get clumsy when incorporating into scripts.

As I mentioned in my previous post, one goal we have for this year is to add a Dhall implementation of the language server protocol:

… and as I’m writing up this post a contributor has already done some initial groundwork on this (see the above issue).

Other language features

The most commonly requested language feature was (bidirectional) type inference:

Type inference

Also, having to literally specify type parameters to everything forces you to name and define all sorts of intermediate types, which makes things very messy. I don’t know enough about Dhall to know if this is possible/desirable to avoid

Lack of gradual typing for type arguments is the biggest deal breaker - it breaks the whole “gradual typing” story

Easing migrating existing large structures to Dhall by ensuring gradual typing is viable

I don’t expect Dhall to get bidirectional type inference at this point. The two main reasons are:

  • This is harder to standardize and port to multiple languages
  • This would be very disruptive at a point where we are trying to stabilize the language

So in my opinion that ship has sailed, although I no longer have veto power over proposed changes so there is still always the possibility that somebody puts forth a compelling proposal for type inference that proves me wrong.

Building maps in yaml/json with a dynamic set of keys is a challenge. The other side of the fence is less strongly typed and so they define fields that may or may not be present, or are keyed on userland values. This sucks but if dhall is going to target tools that use yaml/json, there needs to be a way with good ergonomics to build those values. …

I do plan on adding better support for working with weakly-typed dictionaries. As I mentioned earlier, I plan to port one of the more difficult ops-related configuration formats to Dhall to guide the language design and this will inform how I propose to support these weakly-typed dictionaries.

Turing completeness

I don’t plan on making the language Turing-complete. The absence of Turing completeness is Dhall’s sine qua non.

The ability to to simple math on anything other than Nats. Being able to add or multiply two Doubles would vastly increase Dhall’s usefulness to me.

I’d recommend opening an issue for this if you are interested in Double arithmetic. There is a bit to discuss here that won’t fit in this post:

  1. Simpler syntax for input data to write configuration files. 2. Recursion.

I’m not sure if (1) would be solved by the planned support for importing from JSON (as JSON syntax is still a bit clumsy for people to author in my opinion), but at the moment there aren’t any other plans along those lines.

I also don’t think Dhall will get native language support for recursion (or at least not anytime soon). Recursion will likely remain a design pattern, as described in this document:

… rather than a language feature.

Preserving comments

Two people specifically requested support for fixing a very specific bug, which is that the Haskell implementation swallows all comments except the leading comment when formatting code:

Preserving comments in the output of dhall lint.

https://github.com/dhall-lang/dhall-haskell/issues/145

I hope to get to this soon, because I understand what a pain this is (it bites me, too).

Packaging

Better “package” story. …

Packaging/versioning. Importing from URL makes builds prone to flakiness, local caching helps but is insufficient. Currently using Make to clone repositories at given tags and import locally, which can be a hassle since Make won’t fetch transitive dependencies for the types/expressions that are cloned. Expressions which import types off local disk refer to relative locations, which are prone to breakage if those locations change i.e. due to refactoring. Some kind of standard which allows for a Cargo-style definition of imported types, locked to a version, with a separate lifecycle for resolving imports would improve stability.

I’m a bit resistant to this feedback (although I might have misunderstood it). I think the traditional packaging model adds a lot of unnecessary complexity to distributing code. I view Dhall as the “Bash of functional programming”, where the way you author code is you just save a file and the way you consume code is you refer directly to the file you want (except that Dhall fixes some mistakes Bash made in this regard).

The other reason I question the traditional packaging model is that I don’t see the value that it adds above hosting Dhall code on GitHub or GitLab, especially given that Dhall can import from URLs and resolve relative references correctly.

However, I do think there is value in making Dhall packages easier to discover, browse, and document (i.e. like Hackage for the Haskell ecosystem).

Possibly old interpreters

… less verbose support for using union types more easily

Making it simpler to declare Sum types

I believe these respondents may be using an older version of the interpreter because this should be addressed in the latest release of the language and Haskell interpreter. See this page for more details:

However, reading this makes me realize that next year I should add a question asking respondents what implementation and what version they use. That would also help me gauge how long to support deprecated features (such as the constructors keyword).

Funding

“Books and/or merchandise” was the clear leader for funding mechanism, although I suspect that’s primarily because people want a book (paid or not) based on the overwhelming feedback in favor of documentation:

  • Books and/or merchandise - 24 (52.2%)
  • Crowdfunding (recurring) - i.e. Patreon - 18 (39.1%)
  • Donation button - 14 (30.4%)
  • Project bounties - 14 (30.4%)
  • Crowdfunding (one-time) i.e. Kickstarter - 9 (19.6%)
  • Opening PRS myself :) - 1 (2.2%)
  • Consulting - 1 (2.2%)
  • Open source sponsorship - 1 (2.2%)
  • Company donation for time/moral license - 1 (2.2%)

Note that I also plan to apply for grants for open source work. I didn’t list that as a funding option because my mind was already made up on that.

The funding mechanism that surprised me the most was “Donation button”. I thought that was something that people didn’t really do any more (i.e. not “hip”), but it was tied for third-most popular funding mechanism.

I didn’t list consulting as one of the funding mechanisms (which one respondent had to write in), because I’ve heard from a few sources that consulting can create perverse incentives because simplifying things means fewer billable hours. However, on more reflection I think it might be worth getting corporate sponsorship for performance improvements to the language (as previously mentioned), because that’s less likely to create the wrong incentives or lead to undue corporate influence over the language evolution.

Conclusions

This post doesn’t include the comments of praise in the interest of modesty, but I did read them all and really appreciated them! 馃檪

Hopefully this gives people an idea of where my current priorities are at and also helps others understand how they might be able to contribute to the Dhall ecosystem, too!

Also, if this post is the first you are hearing about the survey, you can still complete the survey and I’ll read your response even if it won’t be summarized in this post. I still get an e-mail notification for each new submission.

You can also use the Dhall language’s issue tracker to provide feedback of any kind, too. For more details on how to discuss, propose, or contribute changes, see the following guide:

Monday, February 11, 2019

Haskell command-line utility using GHC generics

cli-twitter

Today, Justin Woo wrote a post about writing a simple Haskell command-line utility with minimal dependencies. The utility is a small wrapper around the nix-prefetch-git command.

In the post he called out people who recommend overly complex solutions on Twitter:

Nowadays if you read about Haskell on Twitter, you will quickly find that everyone is constantly screaming about some “advanced” techniques and trying to flex on each other

However, I hope to show that we can simplify his original solution by taking advantage of just one feature: Haskell’s support for generating code from data-type definitions. My aim is to convince you that this Haskell feature improves code clarity without increasing the difficulty. If anything, I consider this version less difficult both to read and write.

Without much ado, here is my solution to the same problem (official Twitter edition):

{-# LANGUAGE DeriveAnyClass        #-}
{-# LANGUAGE DeriveGeneric         #-}
{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE OverloadedStrings     #-}
{-# LANGUAGE RecordWildCards       #-}

import Data.Aeson (FromJSON, ToJSON)
import Data.Text (Text)
import Options.Generic (Generic, ParseRecord)

import qualified Data.Aeson
import qualified Data.ByteString.Lazy
import qualified Data.Text.Encoding
import qualified Data.Text.IO
import qualified Options.Generic
import qualified Turtle

data Options = Options
    { branch   :: Bool
    , fetchgit :: Bool
    , hashOnly :: Bool
    , owner    :: Text
    , repo     :: Text
    , rev      :: Maybe Text
    } deriving (Generic, ParseRecord)

data NixPrefetchGitOutput = NixPrefetchGitOutput
    { url             :: Text
    , rev             :: Text
    , date            :: Text
    , sha256          :: Text
    , fetchSubmodules :: Bool
    } deriving (Generic, FromJSON)

data GitTemplate = GitTemplate
    { url    :: Text
    , sha256 :: Text
    } deriving (Generic, ToJSON)

data GitHubTemplate = GitHubTemplate
    { owner  :: Text
    , repo   :: Text
    , rev    :: Text
    , sha256 :: Text
    } deriving (Generic, ToJSON)

main :: IO ()
main = do
    Options {..} <- Options.Generic.getRecord "Wrapper around nix-prefetch-git"

    let revisionFlag = case (rev, branch) of
            (Just r , True ) -> "--rev origin/" <> r
            (Just r , False) -> "--rev " <> r
            (Nothing, _    ) -> ""

    let url = "https://github.com/" <> owner <> "/" <> repo <> ".git/"

    let command =
            "GIT_TERMINAL_PROMPT=0 nix-prefetch-git " <> url <> " --quiet " <> revisionFlag

    text <- Turtle.strict (Turtle.inshell command Turtle.empty)

    let bytes = Data.Text.Encoding.encodeUtf8 text

    NixPrefetchGitOutput {..} <- case Data.Aeson.eitherDecodeStrict bytes of
        Left  string -> fail string
        Right result -> return result

    if hashOnly
    then Data.Text.IO.putStrLn sha256
    else if fetchgit
    then Data.ByteString.Lazy.putStr (Data.Aeson.encode (GitTemplate {..}))
    else Data.ByteString.Lazy.putStr (Data.Aeson.encode (GitHubTemplate {..}))

This solution takes advantage of two libraries:

  • optparse-generic

    This is a library I authored which auto-generates a command-line interface (i.e. argument parser) from a Haskell datatype definition.

  • aeson

    This is a library that generates JSON encoders/decoders from Haskell datatype definitions.

Both libraries take advantage of GHC’s support for generating code statically from datatype definitions. This support is known as “GHC generics”. While a bit tricky for a library author to support, it’s very easy for a library user to consume.

All a user has to do is enable two extensions:

… and then they can auto-generate an instance for any typeclass that implements GHC generics support by adding a line like this to the end of their data type:

You can see that in the above example, replacing SomeTypeClass with FromJSON, ToJSON, and ParseRecord.

And that’s it. There’s really not much more to it than that. The result is significantly shorter than the original example (which still omitted quite a bit of code) and (in my opinion) easier to follow because actual program logic isn’t diluted by superficial encoding/decoding concerns.

I will note that the original solution only requires using libraries that are provided as part of a default GHC installation. However, given that the example is a wrapper around nix-prefetch-git then that implies that the user already has Nix installed, so they can obtain the necessary libraries by running this command:

… which is one of the reasons I like to use Nix.

Wednesday, January 16, 2019

Dhall - Year in review (2018-2019)

dhall-2018

The Dhall configuration language is now two years old and this post will review progress in 2018 and the future direction of the language in 2019.

If you’re not familiar with Dhall, you might want to visit the official website for the language, which is the recommended starting point. This post assumes familiarity with the language.

Also, I want to use this post to advertise a short survey that you can take if you are interested in the language and would like to provide feedback:

Progress in 2018

This section will review the highlights of what we accomplished over the last year. These highlights are not exhaustive and I focus on improvements that might encourage people to revisit the language if they were on the fence a year ago.

If you’re already familiar with recent progress in the language and you are more interested in where the language is going then you can jump to the Future direction section.

New language bindings

Several contributors stepped up to the plate to begin three new actively-maintained language bindings to Dhall.

Of these three the Clojure bindings are the ones closest to completion:

The Clojure bindings are sufficiently close to completion that they currently get an official vote on proposed changes to the language standard, giving them an equal voice in the language evolution process.

This is a complete reimplementation of the language entirely in Clojure that allows you to marshal Dhall expressions, including Dhall functions, directly into Clojure:

The Clojure bindings pave the way for making the Dhall configuration language a first class citizen on the JVM.

Additionally, two other language bindings have gotten pretty far along:

These latter two language bindings haven’t announced yet as they are works in progress, but I still wanted to recognize their work so far.

Also, I want to mention that adding a conformance test suite (thanks to Fabrizio Ferrai) helped drive parallel implementations by providing implementors with a tangible measure of progress towards the goal of 100% standard coverage.

Haskell - Cabal

Thanks to the work of Oliver Charles you can generate .cabal files with Dhall by using dhall-to-cabal.

This part of the project’s README sold me on the motivation for doing so:

We can go beyond Cabal files. If Cabal is a domain specific language for building Haskell projects, what does a domain specific language for building Haskell web applications look like? Does the separate of library, executable, and test-suite make sense here? Maybe we’d rather:

… and have this take care of some other details.

When you think about this it makes perfect sense: Haskell programmers use Cabal/Hackage to package and distribute Haskell code, but then what do they use to package and distribute “Cabal code”? The answer is a language like Dhall that builds in its own code distribution mechanism instead of relying on a separate build tool. This closes the loop so that you don’t need to maintain a growing tower of build tools as your project expands.

I’m pretty sure Cabal was the first “heavy duty” configuration format tested with Dhall because this project prompted the first swell of feature requests related to interpreter performance and usability improvements for working with giant schemas.

Also, the dhall-to-cabal project includes the entire Cabal schema encoded as a Dhall type, which you can find here:

This comes in handy when you want a systematic listing of all Cabal configuration features. If you have the dhall interpreter installed you can also view the normal form of the schema in all its glory by running:

You can also migrate an existing project using cabal-to-dhall, a tool which converts a .cabal file to the equivalent .dhall file.

Eta

Javier Neira with the support of TypeLead added Dhall as a supported file format for configuring Eta packages by building on top of the dhall-to-cabal project.

That project has also produced work-in-progress Eta and Java bindings to Dhall bindings along the way by using Eta to compile the Haskell implementation of Dhall to the JVM. When those are complete you will have yet another option for using the Dhall configuration language on the JVM

If you are interested, you can follow the progress on those bindings via this GitHub issue:

Kubernetes

Dhall is commonly used for ops, and the first project to systematically integrate Dhall into a widely used ops tool is the dhall-kubernetes project, thanks to the work of Arian van Putten, Fabrizio Ferrai, and Thomas Scholtes.

I’ve never used Kubernetes, but everybody tells me that Kubernetes configurations are large, repetitive, and error-prone YAML files, which are the perfect use case for Dhall.

PureScript - Spago

The Spago project builds on top of psc-packages to assemble a PureScript package set to build using Dhall as the configuration format.

This tool takes advantage of Dhall’s import system so that the package set can be split over multiple files, which you can see in the top-level package set here:

… and users can easily import that and easily override packages locally without dealing with the headache of rebasing their local changes whenever the upstream package set changes.

Complete language standard

Last year I promised to upstream all features from the Haskell implementation into the language standard, since at the time a few import-related features were implementation-defined. This was a top priority because I didn’t want to treat other language bindings as second-class citizens.

This year we successfully standardized everything, meaning that there should no longer be any implementation-defined features. Additionally, all new functionality now begins with a change to the standard followed by a change to each implementation of the language, meaning that the Haskell implementation is no longer treated as a distinguished implementation.

Unsigned Natural literals

Earlier this year Greg Pfeil proposed to fix the obligatory + sign preceding Natural number literals, which bothered a lot of newcomers to the language. He proposed require a leading + for non-negative Integers instead of Natural numbers.

We knew this would be a highly breaking change, but we were all tired of the + signs which littered our code. The Natural type is much more natural to use (pun intended) than the Integer type, so why not optimize the syntax for Natural numbers?

So we made the change and now instead of writing an expression like:

… you instead write:

This change also improved code comprehension, because before this change an expression like this:

… could be misconstrued as adding f to various numbers, but after this change:

… the reader can more easily discern that f is being applied as a function to numeric arguments.

Type synonyms

Previously, users couldn’t create new types using let expressions and had to work around this limitation by using the import system to reuse types, like this:

Now users can define new types inline within the same file using an ordinary let expression:

Simpler Optional literals

Optional literals used to resemble lists, like this:

Now you can use Some and None instead, like this:

In particular, a present Optional literal no longer requires a type since Some can infer the type from the provided argument. This simplifies the common idiom of overriding an absent Optional value within a record of defaults:

The old list-like syntax is still supported but is deprecated and will be dropped. dhall lint will also automatically replace the old list-like Optional literals with their new Some/None equivalents.

Union constructors

Unions used to be one of the major pain points when using the language, due to having to specify all alternatives for a union literal, like this:

My first attempt to improve this introduced the constructors keyword which took a union type as an argument and returned a record of functions to create each constructor. This changed the above code example to:

However, this initial solution introduced two new problem:

  • A lot of constructors-related boilerplate at the beginning of Dhall files
  • Performance issues due to these large intermediate records of constructors

A follow-up change resolved both issues by overloading the . operator to also access constructor functions directly from a union type (as if it were already a record), like this:

This solved both the performance issue (by eliminating the need for an intermediate record of constructors) and eliminated the constructors keyword boilerplate. Also, this simplification plays nicely with the auto-complete support provided by the dhall repl since you can now auto-complete constructors using the . operator.

Import caching

In 2017 Dhall added support for semantic integrity checks, where you tag an import with a SHA256 hash of a standard binary encoding of an expression’s normal form. This integrity check protects against tampering by rejecting any expression with a different normal form, guaranteeing that the import would never change.

Several astute users pointed out that you could locally cache any import protected by such a check indefinitely. Even better, the SHA256 hash makes for a natural lookup key within that cache.

We standardized and implemented exactly that idea and now any import protected by an integrity check is permanently cached locally using the standard directory prescribed by the XDG Base Directory Specification.

For example, you can now import the entire Prelude as a package protected by an integrity check:

The first time you resolve the Prelude the import may take a bit (~7 seconds on my machine) to fetch the entire package, but the normal form is then stored locally in a 5 KB file:

… and then subsequent attempts to import the same Prelude resolve much more quickly (~80 ms on my machine).

This means that you can now cheaply import the entire Prelude in every file instead of separately importing each function that you use.

Alternative imports

The language now provides support for fallback imports using the ? operator if import resolution fails.

For example, you can use this feature to import an environment variable if present but gracefully fallback to another value if absent:

Or you can use the ? operator to provide alternative locations for obtaining an imported expression:

Multi-let expressions

You can now define multiple values within a single let expression instead of nesting let expressions. In other words, instead of this:

… you can now write this:

dhall lint will also automatically simplify any code using the old nested let style to use the new “multi-let” style.

Statically linked executables

The Haskell implementation of Dhall strives to be like the “Bash of typed functional programming”, but in order to do so the implementation needs to small, statically linked, and portable so that sysadmins don’t object to widely installing Dhall. In fact, if the executable satisfies those criteria then you don’t even need your sysadmin’s permission to try Dhall out within your own workspace.

Niklas Hamb眉chen made this possible through this through his general-purpose work on fully static Haskell executables built using Nix. Now Dhall’s continuous integration system produces small (< 3 MB) Linux executables that have no dependency footprint whatsoever.

Major performance improvements

The Haskell implementation of Dhall has made dramatic strides in performance improvements over the last year, motivated by projects with very large schemas, such as:

… as well as Formation’s internal use of Dhall which has led to them upstreaming many performance improvements to handle large Dhall programs.

Thanks to the work of Fintan Halpenny, Greg Pfeil, @quasicomputational, and others the Haskell implementation is between 1 to 3 orders of magnitude faster than it was a year ago, depending on the configuration file that you benchmark.

We’re also not done improving performance! We continue to improve as new projects continue to stretch the boundaries of what the language can do.

Type diffs

Large projects like these also led to usability improvements when working with gigantic types. The Haskell implementation now displays concise “type diffs” whenever you get a type mismatch so that you can quickly narrow down the problem no matter how much your configuration schema grows. This works no matter how deeply nested the error is.

For example, the following contrived example introduces four deeply nested errors in a gigantic schema (where the type is over 6000 lines long) and the error message still zeroes in on every error:

dhall repl

The Haskell implementation also added a REPL contributed by Oliver Charles that you can use to interactively interpret Dhall code, including sophisticated auto-completion support contributed by Basile Henry:

The REPL comes in handy when exploring large values or types, as illustrated by the dhall-nethack tutorial which uses the REPL:

dhall lint

The Haskell implementation also provides a useful dhall lint subcommand that you can use to not only format code but to also automatically improve the code in non-controversial ways.

For example, dhall lint will automatically remove unused let bindings and will simplify nested let expressions to instead take advantage of the newest multi-let feature.

dhall resolve --dot

Basile Henry also contributed support for visualizing the dependency tree of a Dhall expression like this:

The following tweet illustrates how to use this feature along with example output:

dhall freeze

Thanks to Tobias Pflug you can also automatically take advantage of Dhall’s semantic integrity checks using the dhall freeze subcommand. This command fetches all imports within a Dhall expression and then automatically tags all of them with semantic integrity checks.

For example:

dhall-lang.org

A while ago, Neuman Vong advised me that if you want your open source project to take off, you need a logo, a website, and a live demo in the browser.

So I took that advice to heart and now Dhall has all three! You can try out the language live in your browser by visiting:

This allows people to “try before they buy” and the site links to several other useful resources, such as the …

Dhall wiki

The Dhall wiki contains several useful educational resources for learning the language. The organization of the wiki closely follows the guidelines from this handy post on writing documentation:

The main thing that is missing is to migrate the Haskell tutorial into a language-agnostic tutorial.

Twitter account

You can also now follow the official Twitter account for the language:

This account regularly posts news and tips about the language and ecosystem that you can use to stay abreast of recent progress.

Switch from IPFS to GitHub

Early on in the language history we used IPFS to distribute the Dhall Prelude, but due to reliability issues we’ve switched to using GitHub for hosting Dhall code.

There’s even a convenient link you can use to browse the Prelude:

Future direction

Phew! That was a lot to recap and I’m grateful to all the contributors who made that possible. Now we can review where the language is going.

First, I’m no longer benevolent dictator-for-life of the language. Each new reimplementation of the language gets a vote on the language standard and now that the Clojure implementation of Dhall is essentially complete they get an equal say on the evolution of the language. Similarly, once the PureScript bindings and Python bindings are close to complete they will also get a vote on the language standard, too.

However, I can still use this post to outline my opinion of where the language should go.

Crossing the Chasm

A colleague introduced me to the book Crossing the Chasm, which heavily influenced my approach to designing and marketing the language. The book was originally written for startups trying to gain mainstream adoption, but the book also strongly resonated with my experience doing open source evangelism (first for Haskell, and now Dhall).

The book explains that you need to first build a best-in-class solution for a narrowly-defined market. This in turn requires that you think carefully about what market you are trying to address and strategically allocate your limited resources to address that market.

So what “market” should Dhall try to address?

YAML

One of the clearest signals I’ve gotten from users is that Dhall is “the YAML killer”, for the following reasons:

  • Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors

  • Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard

  • Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for “brownfield” deployments

Does that mean that Dhall is clearly the best-in-class solution for people currently using YAML?

Not quite. The key thing Dhall is missing for feature parity with YAML is a wide array of native language bindings for interpreting Dhall configuration files. Many people would prefer to use Dhall without having to invoke an external executable to convert their Dhall configuration file to YAML.

This is one of the reasons I’ve slowed down the rate of evolution of the standard so that many of the new language bindings have an opportunity to implement the full standard. Also, I expect that once more language bindings have votes on the standard evolution that will further stabilize the language since new features proposals will have a higher bar to clear.

That’s not to say that we will freeze the language, but instead we will focus on strategically spending our “complexity budget” on features that help displace YAML. If we spend our complexity budget on unrelated features then we will increase the difficulty of porting Dhall to new languages without addressing the initial use case that will help Dhall gain mainstream traction.

JSON integration

One of YAML’s features is that all JSON is also valid YAML, by definition. In fact, some people use YAML just for the fact that it supports both JSON and comments.

This suggests that Dhall, like YAML, should also natively support JSON in some way. Dhall’s issue tracker contains a few issues along these lines and the one I would most like to see completed this year is adding support for importing JSON files as Dhall expressions:

Editor support

Another thing Dhall is missing compared to YAML is widespread editor support. This is why another one of my goals for this year is to create a Dhall language server so that any editor that supports the language server protocol (basically all of them) would get Dhall support for free.

Ops

We can actually narrow down Dhall’s “market” further if we really want to be selective about what we work on. Dhall has also grown in popularity for simplifying ops-related configurations, providing several features that ops engineers care about:

  • Strong normalization

    Ops commonly suffers from the dilemma that too much repetition is error prone, but too much abstraction is also error prone if readers of the code can’t effectively audit what is going on. One of Dhall’s unique features is that all code is strongly normalizing, meaning that every expression can be reduced to an abstraction-free normal form. This is made possible by the fact that Dhall is not Turing-complete (another feature favored by Ops).

  • Absolute type safety

    Ops engineers care about reliability since they maintain the software that the rest of their company relies on and any outages can have devastating effects on both the product and the productivity of other engineers.

    This is one of the reason for the Ops flight from Turing-complete languages to inert configuration files like JSON/YAML because Turing-complete languages give you the tools to shoot yourself in the foot. However, Dhall strikes a balance between being programmable while still not being Turing-complete and having a type system with no escape hatches, so you’re incapable of shooting yourself in the foot.

  • Built-in support for importing code

    Another reason that Ops people hate programmable configuration files is that the programming language they pick typically comes with an external build tool for the language that adds one more layer to the tower of build tools that they have to maintain. Now they’ve just replaced one problem (a repetitive configuration file for their infrastructure) which a new problem (a repetitive configuration file for the build tool for the programming language they used to reduce the original repetition).

    Dhall solves this problem well by providing built-in language support for importing other code (similar to Bash and Nix, both also heavily used for Ops use cases). This means that Dhall provides a solid foundaton for their tower of automation because they don’t need to introduce another tool to support a growing Dhall codebase.

  • Dhall displaces YAML well

    YAML configuration files are incredibly common in Ops and “infrastructure as code”. Example tools that use a YAML configuration are:

    • Kubernetes
    • Docker Compose
    • Concourse
    • Ansible
    • Travis

    YAML is so common that Ops engineers sometimes half-jokingly refer to themselves as “YAML engineers”.

    As already mentioned above, Dhall provides a sane alternative to YAML.

We’ve already seen one Dhall integration for an Ops tool emerge last year with the dhall-kubernetes project and this year I hope we continue along those lines and add at least one more Ops-related integration.

I think the next promising integration is the dhall-terraform project which is still a work in progress that would benefit from contributions.

Funding

Finally, I would like to experiment with various ways to fund open source work on Dhall now that the language has a growing userbase. In particular, I’d like to fund:

  • additional language bindings
  • better editor support
  • adding CI support for statically linked Windows and OS X binaries
  • packaging Dhall for various software distributions (i.e. .rpm/.deb)

… and I’d like to provide some way to reward the work of people who contribute beyond just acknowledging their work in posts like this one.

That’s why one of the survey questions for this year asks for suggestions on what would be the most appropriate (non-proprietary) funding model for that sort of work.

Conclusion

Hopefully that gives people a sense of where I think the language is going. If you have any thoughts on the direction of the language this would be a good time to take the survey:

Like last year, I will follow-up a month from now with another post reviewing the feedback from the survey.