Planet Caml [EN]

Our .NET PDF Toolkit, in 100% F#, cross-compiled with OCaml is now available, starting at £495. It does everything the command line tools do, and more, but is usable natively from VB.NET, C# and ASP.NET.

At the same time, we’ve updated the Command Line Tools (written in pure OCaml) with bug fixes and new features, and now provided for Solaris 10 Sparc and Intel out of the box.

The codebase of about 40,000 lines ended up requiring only 30 points at which conditional compilation was needed to cover the differences between F# and OCaml, though plenty of code had to be modified to compile in both environments in a way that didn’t require conditional compiltion.

There will be a new release of CamlPDF soon, which will have many new features, and which will cross-compile with F# out of the box.

Last time we saw how Camlp4 represents OCaml abstract syntax trees. We also saw some small examples of using Camlp4 quotations to get the AST corresponding to a piece of OCaml code. In this post we'll build a simple code-generating tool that uses quotations in more complicated ways.

The tool takes constructor names on stdin, one per line. It prints an OCaml module on stdout containing a type definition with the given constructors, a function to_string that converts a constructor of the type to the string of its name, and a function of_string that does the reverse. (Not the most useful program ever written.) We'll go through it line by line:

open Camlp4.PreCast

let _loc = Loc.ghost in

We start out with some boilerplate. Recall that Camlp4.PreCast gets us the Ast module containing the AST datatypes (along with some helper functions we'll see below); and that we need a location bound to _loc since the expanded quotations mention it.

let cons =
 let rec loop () =
 try
 match read_line () with
 | "" -> []
 | c -> c :: loop ()
 with End_of_file -> [] in
 loop () in

This part just reads lines from stdin until EOF and puts them in a list, nothing to do with Camlp4.

Printers.OCaml.print_implem
<:str_item<

Now we're into the meat. We've begun a str_item quotation (recall that this is a module body), and we're passing it to a function that will pretty-print it to stdout. The module Printers.OCaml comes from camlp4/Camlp4/Printers/OCaml.ml. (This module is called to pretty-print OCaml code when you run Camlp4; toward the bottom you can see some useful command-line options.)

type t =
 $Ast.TySum (_loc,
 Ast.tyOr_of_list
 (List.map
 (fun c -> <:ctyp< $uid:c$ >>)
 cons))$

It's a bit confusing with the interjected explanations, but we are still in the quotation (up to the >> at the end of the program). This part generates the variant type definition. If the input is the lines "Foo", "Bar", "Baz", we want to generate type t = Foo | Bar | Baz .

This might be a good time to run the above type definition through Camlp4 to see what the AST looks like. You'll see that the body of the definition begins with Ast.TySum, then contains the branches of the variant collected by Ast.TyOr's. Each branch is an Ast.TyId with an Ast.IdUid inside containing the constructor name. We want to come up with the same thing for an arbitrary list of constructor names.

Reading from the inside out, we take the list of constructor names and map a ctyp quotation over it. In this quotation we see an antiquotation. Antiquotations let you fill in code so you can use a quotation as a template. Here the tag uid: turns the string c into an Ast.IdUid inside an Ast.TyId. So we have a list of Ast.TyId (Ast.IdUid "Foo")'s.

Now we call a helper function Ast.tyOr_of_list. There are a bunch of these in camlp4/Camlp4/Struct/Camlp4Ast.mlast, which is where the Ast module comes from. This one collects the elements of the list with Ast.TyOr. (As we saw with Ast.StSem/Ast.StNil, the AST doesn't use regular OCaml lists, but builds list-like things out of data constructors.)

Finally we wrap a Ast.TySum around the whole thing, and use an antiquotation to insert it as the body of the type definition. (Here no tag is needed on the antiquotation; this is usually the case if you're inserting an AST rather than a string.)

let to_string = function
 $Ast.mcOr_of_list
 (List.map
 (fun c -> <:match_case< $uid:c$ -> $`str:c$ >>)
 cons)$

let of_string = function
 $let ors =
 Ast.mcOr_of_list
 (List.map
 (fun c -> <:match_case< $`str:c$ -> $uid:c$ >>)
 cons) in
 Ast.McOr(_loc,
 ors,
 <:match_case< _ -> invalid_arg "bad string" >>)$

>>

Now that we've seen how things work, the functions are pretty easy to understand. There are a few new elements: A pattern match (in either a match or function expression) consists of an Ast.McArr for each arrow (which we make with the match_case quotation), collected by Ast.McOr's (which we make with the Ast.mcOr_of_list helper).

The `str antiquotation turns a string into a quoted OCaml string, escaping any special characters.

If you run a sample match through Camlp4 you'll see constructors like Ast.PaId and Ast.PaStr. These are for patterns on the left side of match cases. When we use the match_case quotation we don't have to worry about it; we can use the same antiquotations on the left and right, and the appropriate constructor is parsed out.

That's the whole program. Here's a Makefile for building it:

variant: variant.ml
 ocamlc \
 -pp camlp4of -I +camlp4 \
 -o variant camlp4lib.cma variant.ml
# ocamlfind ocamlc -syntax camlp4o \
 -package camlp4.quotations.o -package camlp4.lib \
 -linkpkg -o variant variant.ml

clean:
 rm -f variant *.cm* *~

Quotations and antiquotations are a slick way to generate OCaml code, but it can be pretty hard to figure out how to use them. The first thing to realize is that you don't have to: anything you can do with Camlp4 you can do by working directly with the AST constructors (although you will find it tedious). As you learn your way around you can replace explicit AST code with quotations.

To access the full power of quotations we'll need to dive into the OCaml parser. But that's going to have to wait until the next post.

It is like I said : it doesn't work and it bothers me. I have to create a little game and I hoped to changed things the way I liked. It is not a major problem but it could be better to fix this problem.

The module and/or file naming system for ocaml has some annoying restrictions that a modern language like ocaml shouldn't run it. Eg today I was trying to create a packed module, and one of the modules to be packed into it has the same ...

Another tricky one I'm afraid. I have an executable which, when compiled with 3.11.0, fails almost immediately on one machine with: Fatal error: out of memory. The same executable operates normally on another machine, and works on all ...

Walking a directory tree with 25000 files took 130 ms (in OCaml), so it's not free by any means. Perhaps fast enough to bother trying though? Would simplify my life. The way how things now work is like this: ...

OCaml Meeting 2009 in Grenoble -- subscription open / More cores / camlp4 vs camlp5 / OCamlSpotter: OCaml compiler extension for source browsing, version 1.0 / LablGL 1.03 / LablGtk 2.12.0

Camlp4 is one of the best things about OCaml, but because it isn't well-documented (particularly so the revised version released with OCaml 3.10 and later), it isn't used as widely as it might be. I use it in my orpc and jslib (part of ocamljs) projects, and to do so I've had to learn it the hard way, by reading the source code and experimenting. (Of course I have also found the Camlp4 wiki and documentation and tutorials covering the old version useful.) This is the first of a projected series of posts containing what I learned, so others can pick it up faster and so I don't forget it myself.

A good place to start is the datatypes representing OCaml abstract syntax trees. Values of these datatypes are produced by parsing, manipulated by syntax extensions, and converted to concrete syntax by pretty-printing. The easiest way to understand ASTs is to get Camlp4 to show you the AST for a piece of OCaml code. Put some code in a file test.ml:

let q = <:str_item< let f x = x >>

then expand it with camlp4of test.ml -printer o. The << >> syntax introduces a Camlp4 quotation, which takes OCaml concrete syntax and replaces it with the corresponding AST values. The :str_item part says which of the mutually-recursive AST types we want to produce; a str_item is something that can appear at the top level of a module body. The invocation camlp4of runs Camlp4 with the quotation module loaded, among others, and the option -printer o means pretty-print the result using standard (or "original") OCaml syntax. The result is:

let q =
 Ast.StSem (_loc,
 Ast.StVal (_loc, Ast.BFalse,
 Ast.BiEq (_loc,
 Ast.PaId (_loc, Ast.IdLid (_loc, "f")),
 Ast.ExFun (_loc,
 Ast.McArr (_loc,
 Ast.PaId (_loc,
 Ast.IdLid (_loc, "x")),
 Ast.ExNil _loc,
 Ast.ExId (_loc,
 Ast.IdLid (_loc, "x")))))),
 Ast.StNil _loc)

Some things to notice here: The expanded value is given by constructors from an Ast module. If you open Camlp4.PreCast at the top of the file (and give the flags -I +camlp4 to ocamlc, or -package camlp4 to ocamlfind ocamlc) you'll get an appropriate Ast module. The full datatype is defined in camlp4/Camlp4/Camlp4Ast.partial.ml in the OCaml source (this file, as with most of Camlp4, is written in the revised syntax).

All the constructors take a _loc argument, which is a value of type Loc.t (when using Camlp4.PreCast this is Camlp4.Struct.Loc). When a source file is parsed in the normal way (not via a quotation) then this argument is the location in the source file where a piece of AST came from. When you're generating code with quotations you have to put something in here, by binding something to _loc before the quotation. For now you can use Loc.ghost, which is just a dummy.

The StSem and StNil constructors are used to collect parts of a module. When you parse ordinary code you'll always get a list: a chain of StSem's with some actual structure item in the first position (after the location) and the tail of the list in the second, terminated by a StNil. If you generate code with nested lists, they'll be flattened out when the AST is pretty-printed. This is convenient when you're building up a complicated quotation, so you don't have to worry about flattening it yourself.

Let's try another one:

let q = <:ctyp< ('a, 'b) foo >>
let r = <:ctyp< 'a 'b foo >>

(here a ctyp is a type expression) which results in

let q =
 Ast.TyApp (_loc,
 Ast.TyApp (_loc,
 Ast.TyId (_loc, Ast.IdLid (_loc, "foo")),
 Ast.TyQuo (_loc, "a")),
 Ast.TyQuo (_loc, "b"))

let r =
 Ast.TyApp (_loc,
 Ast.TyId (_loc, Ast.IdLid (_loc, "foo")),
 Ast.TyApp (_loc, Ast.TyQuo (_loc, "b"),
 Ast.TyQuo (_loc, "a")))

This is a little strange: we have two constructs in the concrete syntax that both turn into TyApp's, one associated to the left and one to the right.

I put in this example (and the StSem/StNil business) to point out that the AST type definition doesn't tell the whole story about how ASTs are put together. Although most of the time you can use quotations, sometimes you need to work directly with the AST: when debugging, when building a very dynamic AST, or when you can't figure out the right quotation mumbo-jumbo. No worries though; it's easy to get Camlp4 to tell you the right AST for a piece of concrete syntax.

Next post I'll cover quotations for OCaml syntax in more detail, and build a simple code generator.

Subject line pretty much says it all - type systems as in ML, Haskell, Scala, etc.

I've tried comp.lang.compilers and comp.lang.functional to no avail. Help me Obi Wan, you're my only hope :-)

If this is not PLT enough a topic, I'll fully understand if the moderator deletes it.

Otherwise, many, many thanks in advance.

Scott

I learned this “trick” some time ago, while reading the source code for the MooTools JavaScript framework (version 1.1, I think). I used this “trick” not so long ago, when developing the Spacetree visualization for the JavaScript InfoVis Toolkit library. When clicking a node, the Spacetree visualization unselects all tree nodes and then selects the nodes in [...]

Functional Pearl: Type-safe pattern combinators, by Morten Rhiger:

Macros still have not made their way into typed higher-order programming languages such as Haskell and Standard ML. Therefore, to extend the expressiveness of Haskell or Standard ML gradually, one must express new linguistic features in terms of functions that fit within the static type systems of these languages. This is particularly challenging when introducing features that span across multiple types and that bind variables. We address this challenge by developing, in a step-by-step manner, mechanisms for encoding patterns and pattern matching in Haskell in a type-safe way.

This approach relies on continuation-passing style for a full encoding of pattern matching. Tullsen's First-Class Patterns relies on a monadic encoding of pattern matching to achieve abstraction over patterns. Given the relationship between CPS and monads, the two approaches likely share an underlying structure.

Abstracting over patterns yields a whole new level of abstraction, which could significantly improve code reuse. The only concern is compiling these more flexible structures to the same efficient pattern matching code we get when the language natively supports patterns. Section 4.9 discusses the efficiency concerns, and suggests that partial evaluation can completely eliminate the overhead.

- A serious bug was found in Mbox.pop. Update recommended. - add 'future' event.

This morning, Alan sends me the link to the subscription form. We are now ready to accept participants for OCaml Meeting 2009.

Last week, I was at the annual CAML consortium meeting. We talked about OCaml Meeting with M. Leroy and other participants. Most of us, think this meeting is a real great way to make OCaml user meet. This can be the start of great projects, just because people realized that a real OCaml community exists. The INRIA seems really enthusiast about the meeting, maybe INRIA team will be directly involved in the organization in 2010.

I hope 2009 meeting will be as good as the former. At least this year, we are doing things quite in advance and the whole organization will be a little bit more "professional".

OCaml Meeting webpage

(none)

It would be useful to be able to pass command line options to libraries linked into the ocaml toplevel. Attached is a patch to add the Arg.Rest option '--' to the ocaml toplevel and function Toploop.get_cli_strings to get the strings ...

when I launch "make -f Makefile.nt world", build stop with the message "win32.c(36) : fatal error C1083: Cannot open include file: 'flexdll.h': No such file or directory". I have also strange message like "process_begin: ...

When using the Num module, it will lock up the program in certain situations. For example: string_of_num (num_of_int max_int */ num_of_int max_int) will freeze the process with 100% CPU usage. On Linux and OS-X 64-bit builds it works, ...

With minor tweaks to the configure script, ocaml 3.11.0 can be compiled on Solaris x86-64. Can the following patch be accepted into future releases? Thanks, yichen.

If a module contains both a sub-module M and a class m, then ocamldoc will produce the same file name (modulo case, which is irrelevant under Windows) for the two components. As an ugly fix, one could simply replace the implementation ...

Type "error" in the former file(s) misses "Inconsistent_implementation" variant etc. I believe it should be updated from the latter one.