A dynamically generated FromJSON instance

Suppose you need to invoke a function (one like decode from pipes-aeson) that has a FromJSON constraint, but the exact parsing method required for the JSON is not fully known until runtime. Perhaps because field names in the JSON have some prefix that is only determined through reading  a configuration file, or by asking the user.

One solution is to use the tools provided by reflection package to dynamically generate the FromJSON instance at runtime, after having constructed the parse function.

The reflection repository already contains an example of a dynamically generated Monoid instance. I have adapted it for the FromJSON case; the code can be found in this gist.

Some tips for defining these kinds of local instances:

• You never ever have to declare a Reifies instance, this is done by the internal machinery of the reflection package, through some dark magic I don't understand, even if it's explained here.
• Instead, you use Reifies as a precondition for the instance you are actually declaring (FromJSON in this case). You are expressing something like "whenever this phantom type s reifies a typeclass dictionary at the type level, I can reflect it back and use the recovered dictionary to implement the methods of my instance".
• The actual value passed to the reflect function is ignored, it only serves to tell the function what type reifies the value we want to reflect. Assuming you have a Reifies s a constraint, you can always pass something like Proxy :: Proxy s to reflect and obtain an a.
• At some point, you will need a small auxiliary function to convince the compiler that the phantom type in your datatype is the same as the phantom type in the Proxy supplied by reify. Otherwise the Reifies constraint won't be satisfied, and your instance won't kick in!
• Notice that, thanks to parametricity, the phantom type in the Proxy supplied by reify can never escape the callback. So be sure to strip from your result any newtype that still carries the phantom type!

See also this answer on Stack Overflow.

Connecting to Denodo Virtual DataPort from Go

Perhaps I can reuse the setup of my previous post for another database connectivity experiment, this time from Go.

I have very little knowledge of Go. It seems like a pragmatic language with good tooling. And there exists a PostgreSQL driver written in pure Go that I can use.

The first step is to install Go and set up the environment. Once ready, the PostgreSQL driver can be installed in the workspace with the command go get github.com/lib/pq.

Then we create a project named github.com/danidiaz/hellopq in the workspace, with the following Go file:

The code was adapted from the one shown in this video. The connection parameters were taken from here.

Executing the program, it seems to work!

But I have a remaining doubt: how to properly handle the nullability of the last_updated column?

Connecting to Denodo Virtual DataPort from Haskell

My employer Denodo has released an "Express" version of its data virtualization platform. The Virtual DataPort component lets you do cool things like joining tables that reside in completely different databases (say, one table in Oracle and another in MySQL). And it comes with a boatload of connectors for a wide range of data sources.

I'm interested in accessing all that data goodness from Haskell. There are a couple of ways of doing it.

One would be to use Virtual DataPort's REST interface. That's a valid option and maybe the subject of another post.

Another is to connect through the ODBC interface on port 9996. As it happens, the wire protocol aims to be compatible with that of PostgreSQL, so maybe I could just use Haskell's popular postgresql-simple library. Let's try it out.

Musicbrainz

First, we need an underlying database instance that we can "wrap" with Virtual DataPort. Let's use the Musicbrainz database, a perennial favorite for testing. We can download it as a virtual machine and run it on Virtual Box. I usually set the guest networking to NAT and only forward PostgreSQL's port, which is 5432.

Virtual DataPort

Then we download and install Denodo Express, start the Virtual DataPort server, and launch the graphical administration tool.

From the administration tool's GUI, we have to import a few tables from the Musicbrainz database. First we create a JDBC data source:

(The Musicbrainz database is called musicbrainz_db, the user/password is musicbrainz/musicbrainz.)

And then import the artist_name and release tables:

Haskell

Ok, now for the Haskell part. postgresql-simple depends on native libraries. In CentOS 7 for example, we'll have to install the package postgresql-devel using yum, before invoking cabal install postgresql-simple.

Once we have the required dependencies, we can dabble with the following snippet (notice that we are connecting to Virtual DataPort, not directly to Musicbrainz):

...and it works! A sweet stream of data flows from the Musicbrainz database, passes through Virtual DataPort and arrives at our Haskell client.

And thanks to the virtualization capabilities of Virtual DataPort, that stream could potentially represent the combined flow of a number of affluents.

Pawn shops, refrigerators, and contravariant functors

Recently, I learned (by reading instance declarations in Haskell) that the composition of contravariant functors is covariant.

In this talk (around 27:30) Erik Meijer tells a colorful analogy for contravariance in subtyping. The analogy involves trashcans, which apparently are contravariant. I wonder if I can find a similar analogy for the case of composition, because I find it difficult to gain an intuition for it.

So, consider bananas. Bananas are a type of fruit.

Consider refrigerators as well. There are different types of refrigerators. Some have a technology advanced enough to freeze any fruit, but some are more modest and can only freeze bananas.

The refrigerators that can freeze any fruit are a subtype of the refrigerators that can freeze bananas. Because, any time you need to freeze a banana, you can freeze it in one of the refrigerators than can handle any kind of fruit.

In other words, refrigerators are contravariant.

Now consider pawn shops. Paws shops are contravariant as well: anything you can sell to a pawn shop that only takes rings, you can sell to a pawn shop that takes any jewelry. The pawn shops that take any jewelry are a subtype of the pawn shops that take rings.

There are also pawn shops for refrigerators. For example pawn shops for refrigerators that can freeze any fruit, or pawn shops for refrigerators that can freeze bananas. (Yes, they are quite specialized pawn shops, but bear with me.)

The pawnbrokers are very meticulous. Before accepting a refigerator for sale, they put inside it a viand of the type associated with the pawn shop, to check that the refrigerator indeed works.

Can you sell to the pawn shop of refrigerators that freeze bananas everything you could sell to the pawn shop of refrigerators that freeze any fruit? The answer is yes. You take the refrigerator that freezes any fruit to the pawn shop, the pawnbroker puts a banana on it, and it gets frozen. Deal!

So pawn shops for refrigerators are covariant on the type of viands that the refrigerators can freeze.

Whew, that was convoluted.

Two new "validation" applicatives

There exists an Either-like Applicative (usually called "Validation") that either collects all the errors, or returns a success. This is different from the Monad instance of Either, that stops at the first error encountered.

The type can be found at the validation package. However, that package incurs on a heavy lens dependency.

Recently, the type has cropped up without the lens dependency in both transformers (as the Errors type synonym) and either (as Validation).

The version in the either package has a more precise type, as it only requires the error values to form a Semigroup, but not necessarily a Monoid. So you can use a NonEmpty to accumulate errors.

conceit 0.2.1.0

I have released a new version of the conceit package with the following changes:

The Applicative instance now doesn't require those ugly (Show e, Applicative e) constraints on the error type.

There's a Monad instance for Conceit. I initially made the >> operator concurrent, like *>, but that was a bad idea. In the end, the Monad instance has sequential operations and the Applicative concurrent ones, like it happens in Haxl.

There are also MonadThrow, MonadCatch and MonadError instances. The first two work throwing and catching exceptions from IO, the last one works more like ExceptT.

A special run function was added for when the error is "impossible":  Conceit Void a -> IO a. This makes easier to use Conceit in place of Concurrently.

The internals of this new version of conceit have been copied from those of Concurrently in Simon Marlow's async package, with modifications to support the new behaviour.

Colchis, yet another JSON-RPC 2.0 client

There's no shortage of JSON-RPC 2.0 libraries on Hackage, as any cursory search will show. I have just added my own, called Colchis.

It is a pipes-based client that makes use of bidirectional pipes. It doesn't have a lot of features, in particular the only supported transport is raw TCP. I think HTTP transport support would be easy to add, but I don't have the need right now. No notifications or batched requests, either.

My aim is to be able to communicate with jsonrpc4j servers in "streaming" mode.

Check the examples folder in the repo for some examples of usage.