ocaml/ocamltest

# Introduction

## Context

The testsuite of the OCaml compiler consists of a series of programs
that are compiled and executed. The output of their compilation and
execution is compared to expected outputs.

Before the introduction of ocamltest, the tests were driven by a set of
makefiles which were responsible for compiling and running the test
programs, and verifying that the compilation and execution outputs were
matching the expected ones.

In this set-up, the precise information about how exactly one test
should be compiled was separated from the test itself. It was stored
somewhere in the makefiles, interleaved with the recipes to actually
compile and run the test. Thus, given one test, it was not easy to
determine exactly how this test was supposed to be compiled and run.

## Purpose

The ocamltest tool has been introduced to replace most of the makefiles
logic. It takes a test program as its input and derives from annotations
stored as a special comment at the beginning of the program the exact
way to compile and run it. Thus the test-specific metadata are stored in
the test file itself and clearly separated from the machinery required
to perform the actual tasks, which is centralized in the ocamltest tool.

## Constraints

It may look odd at first glance to write the tool used to test the
compiler in its target language. There are, however, parts of the
compiler and the standard library that are already tested in a way,
namely those used to compile the compiler itself. Therefore, these
components can be considered more trustworthy than those that have
not yet been used and that's
why ocamltest relies only on the part of the standard library that has been
used to develop the compiler itself.

This excludes for instance the use of the Unix and Str libraries.

# Initial set-up

ocamltest needs to know two things:

1. Where the sources of the OCaml compiler to test are located.
This is determined while OCaml is built. The default location can be
overridden by defining the OCAMLSRCDIR environment variable.

2. Which directory to use to build tests. The default value for this is
"ocamltest" under Filename.get_temp_dir_name(). This value can be
overridden by defining the OCAMLTESTDIR environment variable.

# Running tests

(all the commands below are assumed to be run from OCAMLSRCDIR/testsuite)

From here, one can:

## Run all tests: make all

This runs the complete testsuite. This includes the "legacy" tests
that still use the makefile-based infrastructure and the "new" tests
that have been migrated to use ocamltest.

## Run legacy tests: make legacy

## Run new tests: make new

## Run tests manually

It is convenient to have the following ocamltest script in a directory
appearing in PATH, like ~/bin:

#!/bin/sh
TERM=dumb OCAMLRUNPARAM= /path/to/ocaml/sources/ocamltest/ocamltest $*

Once this file has been made executable, one can for instance run:

ocamltest tests/basic-io/wc.ml

As can be seen, ocamltest's output looks similar to the legacy format.

This is to make the transition between the makefile-based
infrastructure and ocamltest as smooth as possible. Once all the
tests will have been migrated to ocamltest, it will become possible to
change this output format.

The details of what exactly has been tested can be found in
${OCAMLTESTDIR}/tests/basic-io/wc/wc.log

One can then examine tests/basic-io/wc.ml to see how the file
had to be annotated to produce such a result.

Many other tests have already been migrated and it may be useful to see
how the test files have been annotated. the command

find tests -name '*ocamltests*' | xargs cat

gives a list of tests that have been modified and can therefore be used
as starting points to understand what ocamltest can do.

# Migrating tests from makefiles to ocamltest

It may be a good idea to run make new from the testsuite directory before
starting to migrate tests. This will show how many "new" tests there
already are.

Then, when running make new after migrating n tests,
the number of new tests reported by make new should have increased by n.

OCaml's testsuite is divided into directories, each of them
containing one or several tests, which can each consist of one or
several files.

Thus, the directory is the smallest unit that can be migrated.

To see which directories still need to be migrated, do:

find tests -name 'Makefile'

In other words, the directories that still need to be migrated are
the subdirectories of testsuite/tests that still contain a Makefile.

Once you know which directory you want to migrate, say foo, here is
what you should do:

Read foo/Makefile to see how many tests the directory contains and how
they are compiled. If the makefile only includes other makefiles and
does not define any variable, then it means that nothing special
has to be done to compile or run the tests.

You can also run the tests of this directory with the legacy framework,
to see exactly how they are compiled and executed. To do so, use the
following command  from the testsuite directory:

make --trace DIR=tests/foo

(You may want to log the output of this command for future reference.)

For each test, annotate its main file with a test block, i.e. a
comment that looks like this:

(* TEST
  Optional variable assignments and tests
*)

In particular, if the test's main file is foo.ml and the test uses
modules m1.ml and m2.ml, the test block will look like this:

(* TEST
  modules = "m1.ml m2.ml"
*)

And if the test consists of a single file foo.ml that needs to be
run under the top-level, then its test block will look like this:

(* TEST
  * toplevel
*)

Or, if there are two reference files for that test and the name
of one of them contains "principal", then it means the file should
be tested with the top-level, without and with the -principal option.
This is expressed as follows:

(* TEST
  * toplevel
  * toplevel
    include principal
*)

Lines starting with stars indicate which tests to run. If no test is
specified, then the tests that are enabled by default are used,
namely to compile and run the test program in both bytecode and native
code (roughly speaking).

Once your test has been annotated, run ocamltest on it and see
whether it passes or fails. If it fails, see the log file to understand why
and make the necessary adjustments until all the tests pass.

The adjustments will mostly consist in renaming reference files and
updating their content.

Note that there are different types of reference files, those for
compiler output and those for program output.

To make sure the migration has been done correctly, you can compare the
commands used to compile the programs in ocamltest's log file to those
obtained with make --trace. Beware that the commands used to compare an
obtained result to an expected one will not show up in ocamltest's log
file.

Once this has been done for all tests, create a file called "ocamltests"
(mark the final s!) with the names of all the files that
have been annotated for ocamltest, one per line.

Finally, git rm the Makefile and run make new from the testsuite directory
to make sure the number of new tests has increased as expected.