Creating OCaml Code Coverage Reports for Ketrew

Since everyone loves to read about code coverage, we figured we’d describe how we implemented it for a language only 97.3% as popular as JavaScript: OCaml! No matter how functional our approach to implementing Ketrew, our eDSL for managing workflows, at the end of the day the program needs to manage state. From that perspective, code coverage helps you to be sure that you’ve tested all the ways you manage that state, and probably more importantly what paths you’ve implemented but haven’t tested.

OCaml Code Coverage Tools

We considered a couple of options when trying to implement code coverage for OCaml.

1. MLCov works by patching the OCaml compiler. Unfortunately, the latest version (as of 2010-11-24) works against the 3.12 version.
2. ZAMCOV is advertised as a modified OCaml virtual machine that runs your source code and tracks execution. Unfortunately, it also targets version 3.12. Both of these methods seem outdated and do not provide the necessary flexibility with updating versions.
3. Bisect works by first instrumenting the code via Camlp4, and then linking a library that keeps track of the executed code paths. Finally, an executable bisect-report can be used to generate a pretty annotated webpage. Relying on Camlp4 certainly gives us some pause due to the move towards extension points in 4.02, but this seems like the most up to date method.

Bisect

Installing is easy via opam install bisect.

For demonstration if we have example.ml with

let () =
  if Array.length Sys.argv > 1 then
    Printf.printf "Hello %s\n"  Sys.argv.(1)
  else
    Printf.printf "Hello Bisect Coverage!\n"

then

$ camlp4o `ocamlfind query str`/str.cma `ocamlfind query bisect`/bisect_pp.cmo example.ml -o example_instrumented.ml

(Remember that when camlp4 is asked to pipe output it returns the binary OCaml AST representation needed by the compiler) will create:

let () = Bisect.Runtime.init "example.ml"

let () =
  (Bisect.Runtime.mark "example.ml" 2;
   if (Array.length Sys.argv) > 1
   then
     (Bisect.Runtime.mark "example.ml" 1;
      Printf.printf "Hello %s\n" Sys.argv.(1))
   else
     (Bisect.Runtime.mark "example.ml" 0;
      Printf.printf "Hello Bisect Coverage!\n"))

Of course, it is important to be able to control the instrumentation so that production versions do not have this book-keeping. Therefore, we’d like to integrate this capability with our current build tool.

Oasis

Amongst myriad OCaml build tools, we’re using Oasis.

Oasis’s strengths lie in its ability to succinctly represent what you’re trying to build in a way that understands OCaml. If you want to build a library, add a library section; if you want an executable, add an executable section; if you want a test, etc. Oasis does a good job of exposing the appropriate options (such as dependencies, filenames, install flags) for building each of these things, but it is not flexible in how to build these things. Let’s get to the details.

Add a Flag section to the _oasis file to allow you to optionally instrument code:

Flag coverage
  Description: Use Bisect to generate coverage data.
  Default:     false

Unfortunately using this flag in the _oasis file to logically represent two compilation paths is almost impossible. For example, we cannot use BuildDepends.

Adding

  if flag(coverage)
    BuildDepends: bisect
  else
    BuildDepends:

throws up an error: Exception: Failure "Field 'BuildDepends' cannot be conditional". One could create separate build targets for instrumented executables because the Build flag is conditional. But then you would have to duplicate the build chain for all of your intermediary steps, such as libraries, by adding instrumented versions of those. But even if you were successful at that, passing the preprocessing arguments to Ocamlbuild via the XOCamlbuildExtraArgs is settable only in the project scope and you have to pass different arguments to different targets (Library vs Executable).

So for now, add the Flag section: this lets you configure your project with coverage via ocaml setup.ml -configure --enable-coverage by modifying the setup.data text file that is used during compilation.

To perform the instrumentation we’ll drop down a layer into the OCaml build chain.

OCamlbuild

Oasis uses OCamlbuild for the heavy lifting. Besides knowing how to build OCaml programs well and performing it ‘hygienically’ in a separate _build directory, OCamlbuild is also highly configurable with a _tags file and a plugin mechanism via myocamlbuild.ml that supports a rich API. One can write custom OCaml code to execute and determine options; exactly what we need.

The relevant section

let () =
  let additional_rules = function
    | After_rules     ->
      if has_coverage () then
        begin
          let bsdir = Printf.sprintf "%s/%s" (bisect_dir ()) in
          flag ["pp"]
            (S [A"camlp4o"; A"str.cma"; A (bsdir "bisect_pp.cmo")]);
          flag ["compile"]
            (S [A"-I"; A (bsdir "")]);
          flag ["link"; "byte"; "program"]
            (S [A"-I"; A (bsdir ""); A"bisect.cmo"]);
          flag ["link"; "native"; "program"]
            (S [A"-I"; A (bsdir ""); A"bisect.cmx"]);
        end
      else
        ()
    | _ -> ()
  in
  dispatch
    (MyOCamlbuildBase.dispatch_combine
      [MyOCamlbuildBase.dispatch_default conf package_default;
      additional_rules])

performs three functions.

1. It makes sure all the source code passes through the bisect preprocessor (bisect_pp.cmo).
2. Executables (because of the program flag) are linked against the Bisect object file that collects the execution points. The function has_coverage checks that the line coverage="true" is present in setup.data.
3. Lastly, the format of that dispatch makes sure we use ocamlfind when looking for packages.

Reports

We can add some targets to our Makefile to generate reports:

report: report_dir
  bisect-report -I _build -html report_dir bisect*.out

Running against a basic test we get output such as:

Looks like we have more tests to write!