Well, anonymous googlers, today is your day! I'm releasing pipes-2.3 which introduces a new bidirectional pipe type, which I call a Proxy and I've proven the category laws for Proxy composition.
This blog post is not a proper tutorial but rather a meta-discussion of this release. This post discusses context surrounding this release for people who follow iteratee development, so if you just want to see cool examples, then read the Proxy tutorial over at Control.Proxy.Tutorial.
Also, this post is not technically part of my category theory series that I'm writing, but it does fortuitously tie in to it. The Proxy type provides an elegant framework for composing reusable client/proxy/server primitives into powerful applications, so if you started following my blog because of my discussion about compositionality, then I recommend you read the Proxy tutorial.
The Proxy terminology is built on the client-server metaphor, and if you already understand Pipes the following translations will help you map your Pipe intuition onto Proxy terms:
-- Types Pipe -> Proxy Producer -> Server Consumer -> Client Pipeline -> Session -- commands await -> request yield -> respondClients resemble Consumers, except you replace await with request, which provides an argument to upstream:
myClient () = do ... answer <- request argumentServers resemble Producers, except you replace yield with respond. Composition requires a parameter to pass in the first request:
-- +-- 1st request -- | -- v myServer argument = do ...... and every subsequent request is bound to the return value of respond:
myServer argument = do x <- computeSomething argument -- "respond" binds the next argument nextArgument <- respond x myServer nextArgument -- or: myServer = computeSomething >=> respond >=> myServerI provide the foreverK function which abstracts away this common recursion pattern:
-- i.e. forever 'K'leisli arrow foreverK f = f >=> foreverK f myServer = foreverK $ \argument -> do result <- computeSomething argument respond result -- or: myServer = foreverK (computeSomething >=> respond)That looks just like the way you'd write a server's loop: get some argument, compute some result, respond with the result. However, you can do significantly more sophisticated things than just loop.
A Proxy sits between servers and clients. It can query servers on its upstream interface, and respond to clients on its downstream interface:
| Upstream | Downstream | | interface | interface | Proxy arg1 ret1 arg2 ret2 m rAs with Pipes, the intermediate Proxy type is the unifying compositional type which generalizes the endpoint types. Server and Client are just type synonyms around the Proxy type with one of its two ends closed.
You can then compose as many components as you please into a single Session using composition and then use runSession to convert the results back to the base monad:
runSession $ client <-< proxy_1 <-< ... <-< proxy_n <-< serverIn the following sections, I will motivate this upgrade to bidirectional pipes by providing some examples of trivial problems that have embarrassed the entire iteratee community (myself included) up until now.
The simplest example is a file reader. Using any iteratee implementation out there, it is very awkward to specify how many bytes you wish to pull from the upstream source on a request-to-request basis. Most implementations either:
- Hard-code the number of bytes delivered on each request (i.e. conduit/iterIO)
- Initialize the source with a given buffer size and then fix it from that point onward (i.e. enumerator/iteratee)
Unfortunately, the gold standard solution (pushback) is unsatisfactory because it:
- only solves this narrow use case and does not generalize,
- cannot push back portions of input without imposing some sort of Monoid restriction on the iteratee type itself, and
- requires that the user maintain certain invariants to prevent breaking the Category laws.
The next example is interfacing with some server. This is a real-world example from my own work. I've written a protein structural search engine and I've set it up as an RPC service: protein structure goes in, a bunch of search results come out. I'd like to write a Pipes interface to this so I can stream the results coming out of the server, but unfortunately I can't. If I tried, I might do something like this:
searchEngine? :: Pipe Structure [Structure] IO rI can't really accomplish this because Pipes only permit a unidirectional flow of information. I can't both provide the query and receive the results within the same component without resorting to brittle non-compositional tricks like IORefs that defeat the entire point of the iteratee abstraction. However, with Proxys, the solution is incredibly easy:
The input ---------+-------------------+ +- The results | | | v v v searchEngine :: Structure -> Server Structure [Structure] IO r searchEngine = foreverK $ \structure -> do -- "search" might send a network query to the actual server results <- lift $ search structure respond results -- searchEngine = foreverK ((lift . search) >=> respond)Note that this time the query and response occupy the same interface, rather than two opposing interfaces, so I can now hook up a Client to it that send in requests and receive responses within the same block of code.
No other iteratee implementation out there can accomplish this. Instead, they restrict us to using blind sources that don't know what downstream actually wants.
You can also implement imperative-style closures using Proxys. Simply define:
type Closure = Server... and you are good to go! Consider the Python example from the Wikipedia article on closures:
def counter(): x = 0 def increment(y): nonlocal x x += y print(x) return incrementWe can translate this directly into Proxys:
counter :: Int -> Closure Int () IO r counter = counter' 0 counter' x y = do let x' = x + y lift $ print x' y' <- respond () counter' x' y'We can then consume the closure in a structured way using composition:
type Opening = Client -- The opposite of a closure? useClosure :: () -> Opening Int () IO () useClosure () = mapM_ request [1, 7, 1, 1] main = runSession $ useClosure <-< counter... or we can manually peel off individual elements from the closure using runFreeT:
pop :: (Monad m) => a -> Closure a b m r -> m (Maybe (b, Closure a b m r)) pop y = do f <- runFreeT (counter y) case f of Pure _ -> return Nothing Free (Yield x c) -> return $ Just (x, c)Proxy internals are all exposed without compromising any safety, so if you choose not to buy in to the whole composition framework you can always manually deconstruct Proxys by hand and go along your way.
Compositional message passing
As far as I can tell, this is the only bidirectional message passing framework that satisfies the category laws. This guarantees several nice properties:
- The identity laws enforce that composition of components must be completely transparent.
- The associativity law guarantees that each component can be written completely context-free.
When you have compositional components, combining them together is as easy as snapping a bunch of legos together.
Another motivation for this upgrade is finalization. With the ability to send information back upstream, I can now implement bidirectional finalization using ordinary monads and not indexed monads. This will replace Frames, which I will deprecate and either remove or migrate to a separate library.
Pipes are a strict subset of Proxys so if you have existing Pipe code you can replace Control.Pipe with Control.Proxy which provides backwards-compatible definitions for all Pipe primitives and your previous code will still work.
You can understand the relationship between Pipes and Proxys by checking out the type synonym for Pipes provided by Control.Proxy:
type Pipe a b = Proxy () a () bIn other words, a Pipe is a Proxy that never sends any information upstream when it requests input.
There is another advantage of Proxys over Pipes, which is that now it is possible to forbid awaits. The Proxy implementation is highly symmetric and fills a lot of elegance holes that Pipes had.
However, if you love Pipes, never fear, because Control.Pipe will never be deprecated, ever. It provides the simplest iteratee API on Hackage, and I plan to continue to upgrade it with all features compatible with the Pipe type.
One of the surprising results of the bidirectional implementation was that it unifies Kleisli composition and Proxy composition, whose arguments overlap. One thing you will discover the more you program with Proxys is that most useful Proxy components end up being Kleisli arrows and you'll often find that a lot of your code simplifies to the following point-free style:
-- Not that I necessarily recommend writing it this way ((p1 <=< p2 <=< p3) <-< (p4 <=< p5)) <=< (p6 <-< p7)This isn't a coincidence. A very abstract way to understand Proxy composition is that it is just merging lists of Kleisli arrows in a structured way.
I know in the past I've stated that bidirectional information flow does not form a category, so now I'm publicly eating my own words.
There will be two more release in the next two months. The first release will provide the first general mechanism for extending Pipes with your own custom extensions and will include error handling and parsing extensions implemented using this approach.
The second release will provide a second way to customize pipes and will include finalization/reinitialization and stack traces implemented using that approach.