Creating OCaml Code Coverage Reports for Ketrew04 Mar 2015
OCaml Code Coverage Tools
We considered a couple of options when trying to implement code coverage for OCaml.
- MLCov works by patching the OCaml compiler. Unfortunately, the latest version (as of 2010-11-24) works against the 3.12 version.
- ZAMCOV is advertised as a modified OCaml virtual machine that runs your source code and tracks execution. Unfortunately, it also targets version 3.12. Both of these methods seem outdated and do not provide the necessary flexibility with updating versions.
- Bisect works by first instrumenting the code via Camlp4, and then
linking a library that keeps track of the executed code paths.
Finally, an executable
bisect-reportcan be used to generate a pretty annotated webpage. Relying on Camlp4 certainly gives us some pause due to the move towards extension points in 4.02, but this seems like the most up to date method.
Installing is easy via
opam install bisect.
For demonstration if we have
$ camlp4o `ocamlfind query str`/str.cma `ocamlfind query bisect`/bisect_pp.cmo example.ml -o example_instrumented.ml
(Remember that when
camlp4 is asked to pipe output it returns the binary
AST representation needed by the compiler)
Of course, it is important to be able to control the instrumentation so that production versions do not have this book-keeping. Therefore, we’d like to integrate this capability with our current build tool.
Oasis’s strengths lie in its ability to succinctly represent what you’re trying to build in a way that understands OCaml. If you want to build a library, add a library section; if you want an executable, add an executable section; if you want a test, etc. Oasis does a good job of exposing the appropriate options (such as dependencies, filenames, install flags) for building each of these things, but it is not flexible in how to build these things. Let’s get to the details.
Flag section to the
_oasis file to allow you to optionally instrument
Description: Use Bisect to generate coverage data.
Unfortunately using this flag in the _oasis file to logically represent two compilation paths is almost impossible. For example, we cannot use BuildDepends.
throws up an error:
Exception: Failure "Field 'BuildDepends' cannot be
conditional". One could create separate build targets for instrumented
executables because the
Build flag is conditional. But then you would have to
duplicate the build chain for all of your intermediary steps, such as libraries,
by adding instrumented versions of those. But even if you were successful at
that, passing the preprocessing arguments to Ocamlbuild via the
XOCamlbuildExtraArgs is settable only in the project scope and you have to
pass different arguments to different targets (Library vs Executable).
So for now, add the
Flag section: this lets you configure your project with
ocaml setup.ml -configure --enable-coverage by modifying
setup.data text file that is used during compilation.
To perform the instrumentation we’ll drop down a layer into the OCaml build chain.
Oasis uses OCamlbuild for the heavy lifting. Besides knowing how to build OCaml
programs well and performing it ‘hygienically’ in a separate
directory, OCamlbuild is also highly configurable with a
_tags file and a
plugin mechanism via
myocamlbuild.ml that supports a rich API. One can write
custom OCaml code to execute and determine options; exactly what we need.
The relevant section
performs three functions.
- It makes sure all the source code passes through the bisect preprocessor
- Executables (because of the
programflag) are linked against the Bisect object file that collects the execution points. The function
has_coveragechecks that the line
coverage="true"is present in
- Lastly, the format of that dispatch makes sure we use
ocamlfindwhen looking for packages.
We can add some targets to our Makefile to generate reports:
bisect-report -I _build -html report_dir bisect*.out
Running against a basic test we get output such as:
Looks like we have more tests to write!