ocamltest: draft of reference manual (tutorial section)
Sébastien Hinderer
parent
69316099b0
commit
1c0424c80e
|
|
@ -120,6 +120,7 @@ _build
|
|||
/ocamltest/tsl_lexer.ml
|
||||
/ocamltest/tsl_parser.ml
|
||||
/ocamltest/tsl_parser.mli
|
||||
/ocamltest/ocamltest.html
|
||||
|
||||
/otherlibs/dynlink/extract_crc
|
||||
/otherlibs/dynlink/dynlink_platform_intf.mli
|
||||
|
|
|
|||
3
Changes
3
Changes
|
|
@ -33,6 +33,9 @@ Working version
|
|||
|
||||
### Manual and documentation:
|
||||
|
||||
- #9141: beginning of the ocamltest reference manual
|
||||
(Sébastien Hinderer, review by Gabriel Scherer and Thomas Refis)
|
||||
|
||||
### Compiler user-interface and warnings:
|
||||
|
||||
- #9074: reworded error message for non-regular structural types
|
||||
|
|
|
|||
|
|
@ -261,6 +261,15 @@ ocamltest_config.ml: ocamltest_config.ml.in Makefile ../Makefile.config
|
|||
-e 's|@@FUNCTION_SECTIONS@@|$(FUNCTION_SECTIONS)|' \
|
||||
$< > $@
|
||||
|
||||
# Manual
|
||||
|
||||
.PHONY: doc
|
||||
|
||||
doc: ocamltest.html
|
||||
|
||||
ocamltest.html: ocamltest.org
|
||||
pandoc -s --toc -N -f org -t html -o $@ $<
|
||||
|
||||
.PHONY: clean
|
||||
clean:
|
||||
rm -rf ocamltest$(EXE) ocamltest.opt$(EXE)
|
||||
|
|
@ -269,6 +278,7 @@ clean:
|
|||
rm -rf $(cmo_files)
|
||||
rm -rf $(cmx_files)
|
||||
rm -rf $(generated)
|
||||
rm -f ocamltest.html
|
||||
|
||||
ifneq "$(TOOLCHAIN)" "msvc"
|
||||
.PHONY: depend
|
||||
|
|
|
|||
|
|
@ -0,0 +1,745 @@
|
|||
#+STARTUP: showall
|
||||
|
||||
#+title: The ocamltest reference manual
|
||||
#+language: en
|
||||
|
||||
#+HTML_HEAD: <style> body { font-size: 1rem; max-width: 900px; margin: 0 auto; } </style>
|
||||
|
||||
* Introduction
|
||||
|
||||
This is =verbatim= and this is ~code~.
|
||||
|
||||
** What is ocamltest
|
||||
|
||||
ocamltest is a test-driver, that is, a program that can run tests and report
|
||||
their results so that they can be used by a test infrastructure.
|
||||
|
||||
Originally, the tool has been designed specifically to run the integration
|
||||
tests of the OCaml compiler's test suite. However, it has been
|
||||
designed with extensibility in mind and thus has a plugin
|
||||
mechanism that makes it possible to extend it with other tests.
|
||||
|
||||
** Design choices
|
||||
|
||||
*** Programming language and external dependencies
|
||||
|
||||
For a start, one may wonder in which language a test-driver for a compiler
|
||||
should be written. It may indeed seem odd to write a test-driver for a
|
||||
compiler in the language it compiles, since the compiler itself
|
||||
is yet untested and thus not trustworthy.
|
||||
|
||||
It can however be observed that the OCaml compiler is /bootstraped/,
|
||||
meaning that it is itself written in OCaml. A newer version of the
|
||||
compiler can thus be produced from an existing one and the (OCaml)
|
||||
source code of that newer version. Practically, this means that the
|
||||
compiler works at least well enough to recompile itself. This is why we
|
||||
consider that it is okay to write a test-driver like ocamltest in OCaml,
|
||||
as long as it uses only code that has been used to bootstrap the
|
||||
compiler. In particular, this is why we prefer not to rely on any
|
||||
external dependency, not even the libraries included in the compiler
|
||||
distribution such as the =Unix= library.
|
||||
|
||||
*** Test types
|
||||
|
||||
As has been noted above, ocamltest has been developed to run the already
|
||||
existing integration tests of the OCaml compiler's test suite, which
|
||||
were previously run by a set of makefiles. This context explains
|
||||
several design decisions which could otherwise seem rather arbitrary.
|
||||
|
||||
For example, the reason why ocamltest has no support for running unit tests
|
||||
is that there were no such tests in the OCaml compiler's test suite.
|
||||
|
||||
Indeed, the OCaml compiler's test suite is composed mainly of complete
|
||||
programs. In this context, the most current meaning of "testing" a program
|
||||
is that the program needs to be compiled and executed. The test will
|
||||
be considered successful if the program compiles as expected and, when run,
|
||||
returns the expected value.
|
||||
|
||||
Since this scenario is the most frequent one, it was of particular
|
||||
importance to make writing tests of this form as simple as possible.
|
||||
|
||||
However, not all tests fall into the previously described category, so it is
|
||||
also necessary to support not only variations on the previous scenario
|
||||
(compile but do not run, compile with certain options, etc.) but also
|
||||
completely different tests, such as top-level tests, debugger tests,
|
||||
etc.
|
||||
|
||||
To fulfill these requirements and make it as easy as possible to turn a
|
||||
program into a test, it has been chosen to design a Domain-Specific
|
||||
Language (DSL) used to annotate the test program with a
|
||||
=(* TEST *)= block at its top. This block specifies how the test
|
||||
should be performed.
|
||||
|
||||
** Outline of this document
|
||||
|
||||
The next chapter explains through examples how to write simple tests. We
|
||||
then introduce the key concepts used by ocamltest to provide a better
|
||||
understanding of how it works and can be used to write more complex
|
||||
tests. The two last chapters give an in-depth description of the
|
||||
built-in tests and actions and of the tests and actions that are specific
|
||||
to the OCaml compiler.
|
||||
|
||||
* Writing simple tests
|
||||
|
||||
This chapter is a tutorial. It explains how to write simple test
|
||||
programs and also tries to give insights about how ocamltest works. These
|
||||
insights will be deepened in chapter [[#concepts]] where ocamltest is
|
||||
presented in a more abstract and conceptual way.
|
||||
|
||||
We start by explaining how to set-up a proper environment for writing
|
||||
tests. We then show how to turn the traditional "Hello, world!" program
|
||||
into a test and explain how to run it with ocamltest. We continue
|
||||
with a few variations on this test and conclude this chapter
|
||||
with a few other useful tests.
|
||||
|
||||
** Prerequisites for writing tests
|
||||
|
||||
Writing tests requires that the sources of the OCaml compiler for which
|
||||
one wants to write them are downloaded and compiled. The compiler
|
||||
does not need to be installed, though.
|
||||
|
||||
The sources can be downloaded either as an archive, or directly cloned
|
||||
through git, which seems more appropriate in the context of writing ones
|
||||
own tests. Refer to
|
||||
=INSTALL.adoc= (and also to =README.win32.adoc= if you are on Windows) to
|
||||
learn how to get the sources of the OCaml compiler and to compile them.
|
||||
|
||||
In the remainder of this manual, we will assume that the sources of the
|
||||
OCaml compiler have been extracted in the =${OCAMLSRCDIR}= directory (for
|
||||
instance =${HOME}/src/ocaml=) and that you have successfully configured
|
||||
and compiled them as described in =INSTALL.adoc= or =README.win32.adoc=,
|
||||
according to your operating system. The tools and libraries necessary
|
||||
for running tests should also be built. This can be achieved by running
|
||||
the following command from =${OCAMLSRCDIR}=:
|
||||
: make -C testsuite lib tools
|
||||
|
||||
We will also assume that an =ocamltest= command is available in
|
||||
your =PATH=. Although this is not strictly necessary, it is strongly
|
||||
recommended that you set this up because this will simplify test
|
||||
development a lot. This can be achieved e.g. by creating a symbolic
|
||||
link to =${OCAMLSRCDIR}/ocamltest/ocamltest= (or its native
|
||||
counterpart =${OCAMLSRCDIR}/ocamltest/ocamltest.opt=) in a directory that
|
||||
is already in your =PATH=, like =~/bin=.
|
||||
|
||||
** Testing the "Hello, world!" program with the default tests
|
||||
|
||||
*** Turning "Hello, world!" into a useful test program
|
||||
|
||||
Consider the following OCaml implementation of the classical "Hello, world!"
|
||||
program written to a =hello.ml= file:
|
||||
|
||||
: let _ = print_endline "Hello, world!"
|
||||
|
||||
Now assume we would like to make sure that the OCaml compiler can
|
||||
compile this program and that the resulting executable indeed prints the
|
||||
expected output. Here are the required steps to turn the program
|
||||
above into a test usable by ocamltest to verify this:
|
||||
|
||||
1. First, we add a special comment at the very beginning of our =hello.ml=
|
||||
file to make it explicit that it is a test:
|
||||
#+begin_src
|
||||
(* TEST *)
|
||||
|
||||
let _ = print_endline "Hello, world!"
|
||||
#+end_src
|
||||
|
||||
2. We then need to say what the expected outputs are. In our case, we
|
||||
expect that compiling the test produces no output at all and that its
|
||||
execution produces one single line:
|
||||
: Hello, world!
|
||||
To let ocamltest know about this, we create a =hello.reference= file
|
||||
containing the program's expected output -- the line mentioned
|
||||
above. There is nothing special to do for silent compilations
|
||||
since this is what is expected by default and a non-silent
|
||||
compilation would actually cause a test failure.
|
||||
|
||||
3. We can now ask ocamltest to run our test program with the
|
||||
following command:
|
||||
: ocamltest hello.ml
|
||||
|
||||
Running this would produce an output similar to this one:
|
||||
|
||||
#+begin_src
|
||||
... testing 'hello.ml' with 1 (native) => passed
|
||||
... testing 'hello.ml' with 2 (bytecode) => passed
|
||||
#+end_src
|
||||
|
||||
In addition to this output, it may be noticed that the previous
|
||||
command has also created an =_ocamltest= directory whose content will
|
||||
be examined in the next sub-section.
|
||||
|
||||
4. Finally, there is one extra step required if we want our newly created
|
||||
test to be run automatically as part of the OCaml compiler's test suite.
|
||||
We need to move =hello.ml= and =hello.reference= to a directory (say
|
||||
=newtest=) located somewhere
|
||||
below =testsuite/tests= in the compiler's source tree and we
|
||||
need to declare the test. This is done by appending the name of the
|
||||
file containing the =(* TEST *)= comment to an =ocamltests=
|
||||
(mark the final s) file located in the =newtest= directory,
|
||||
alongside the other files relevant to the test. Once this is done,
|
||||
the command
|
||||
: make all
|
||||
executed in the =testsuite= directory of the OCaml compiler' source
|
||||
tree will run all the test suite, which now also includes our own test.
|
||||
|
||||
*** What exactly is going on during the test
|
||||
|
||||
The only thing we know from ocamltest's output when run on =hello.ml= is
|
||||
that it is running two tests named =bytecode= and =native= and that the two of
|
||||
them succeed. This can seem rather uninformative, and in a way it is, but
|
||||
it has to be kept in mind that this information is the one passed by the
|
||||
test-driver (ocamltest) to the test infrastructure. In that respect,
|
||||
this is enough. For us users, though, it is not. That's why
|
||||
ocamltest logs much more details about what is going on in a per-test
|
||||
log file, which should be located in the =_ocamltest/hello/hello.log= file
|
||||
found in the directory where =hello.ml= is.
|
||||
|
||||
Before looking at this log file, notice that it has been created in a
|
||||
test-specific directory. ocamltest creates such a directory for each
|
||||
file it tests and makes sure every file produced as a result of
|
||||
testing this file will be placed in this directory, either directly, or
|
||||
in one of its sub-directories. The latter happens if the test has
|
||||
to be compiled several times, with the same compiler and different
|
||||
command-line options, or with different compilers. In particular,
|
||||
in order to better understand what follows, it may be helpful to
|
||||
remember that =OCaml= actually consists in not less than four compilers:
|
||||
=ocamlc.byte= and =ocamlc.opt= which are the bytecode and native
|
||||
flavors of the bytecode compiler and =ocamlopt.byte= and
|
||||
=ocamlopt.opt= which are the bytecode and native flavors of the native
|
||||
compiler. So, as we will see, ''testing the bytecode compiler''
|
||||
actually involves testing two compilers, and the same goes for ''testing
|
||||
the native compiler''.
|
||||
|
||||
Now that all this has been spelled out, let's examine the log file
|
||||
produced by the test. Although it is too long to be reproduced here,
|
||||
it is recommended to go through it quickly to get an idea of its
|
||||
structure. Here is how it starts:
|
||||
|
||||
#+begin_src
|
||||
Specified modules: hello.ml
|
||||
Source modules: hello.ml
|
||||
#+end_src
|
||||
|
||||
The first line lists the names of the modules the test consists of. The
|
||||
second line is almost similar but if some modules had separate
|
||||
interface files, they would be listed here, too, without the user
|
||||
having to specify them in the list of modules (for each specified =.ml=
|
||||
file, ocamltest looks whether a corresponding =.mli= file exists and, if
|
||||
so, adds it to the list of files to consider).
|
||||
|
||||
The rest of the log file can be split into two parts which are very
|
||||
similar to each other: one for the =native= test and one for the =bytecode=
|
||||
test. Among other things, we learn that each of these tests is composed
|
||||
of nine actions. Before diving into the details of what each of these
|
||||
actions does, let us take this opportunity to introduce a bit of
|
||||
ocamltest terminology. An /action/ is anything that can =pass=, =skip= or
|
||||
=fail=. A =test= is a sequence of such actions. Running
|
||||
a test thus means running each of its actions, in sequence, until all the
|
||||
actions have been run or one of them returns =pass= or =skip=. Whatever
|
||||
the last run action returns, this value will be the result of the whole
|
||||
test.
|
||||
|
||||
To give concrete examples of actions, let's briefly go over the nine ones
|
||||
involved in the =bytecode= test (those for the =native= test are
|
||||
quite similar):
|
||||
|
||||
1. =setup-ocamlc.byte-build-env=:: as its name suggests, this action
|
||||
creates a build environment where a program can be compiled and
|
||||
executed using the =ocamlc.byte= compiler. More precisely, this
|
||||
involves creating a dedicated directory under the test-file specific
|
||||
directory and populating it with the files required by subsequent actions.
|
||||
Depending on what the underlying operating system supports, the files
|
||||
will be either symlinked or copied from the test source directory.
|
||||
|
||||
2. =ocamlc.byte=:: invokes the =ocamlc.byte= compiler in various ways.
|
||||
Here, the test program is compiled and linked, but as we will see
|
||||
later, different behaviors are possible depending on ocamltest
|
||||
/variables/.
|
||||
|
||||
3. =check-ocamlc.byte-output=:: this action compares the compiler's
|
||||
output to a reference file, if one exists. As has been mentioned
|
||||
earlier, the absence of such a reference file specifies that the
|
||||
compiler's output is expected to be empty -- if it is not, this
|
||||
causes a failure of this action and thus of the whole =bytecode=
|
||||
test.
|
||||
|
||||
4. =run=:: now that the program has been successfully compiled, it is
|
||||
run with its standard output and error streams saved to a file.
|
||||
|
||||
|
||||
5. =check-program-output=:: this time it is the output of the program
|
||||
which is compared to a reference file, namely the =hello.reference=
|
||||
file created earlier. So far this comparison succeeds, because the
|
||||
output of the program is identical to the reference file but, as an
|
||||
exercise, one may try to modify the reference file to see how this
|
||||
causes the failure of this action and of the whole =bytecode= test.
|
||||
|
||||
This action concludes the test of the =ocamlc.byte= compiler. We now
|
||||
know that it is able to successfully compile our test program and that
|
||||
the resulting executable runs as expected. The four remaining actions
|
||||
are going to test the =ocamlc.opt= compiler in a similar but not
|
||||
identical way:
|
||||
|
||||
6. =setup-ocamlc.opt-build-env=:: this action is the counterpart of
|
||||
action 1 for the =ocamlc.opt= compiler.
|
||||
|
||||
7. =ocamlc.opt=:: like action 2, this action compiles the test program
|
||||
but with the =ocamlc.opt= compiler.
|
||||
|
||||
8. =check-ocamlc.opt-output=:: again, this action is similar to
|
||||
action 3.
|
||||
|
||||
9. =compare-bytecode-programs=:: here we make sure that the generated
|
||||
executable is correct, but in a different way than for the
|
||||
=ocamlc.byte= compiler. Rather than running it and checking its
|
||||
output, we compare it to the one produced in action 2. Such a check
|
||||
may seem strange, because what it requires is that =ocamlc.byte= and
|
||||
=ocamlc.opt= produce exactly the same binary and not two binaries
|
||||
than perform similarly when they are run, but it has proven useful in
|
||||
the past and has permitted to detect a subtle bug in the compiler.
|
||||
|
||||
** Customizing the default tests
|
||||
|
||||
As has been briefly mentioned, the precise behavior of actions (and
|
||||
thus of tests) may depend on /variables/ whose value can be adjusted in
|
||||
the =(* TEST ... *)= blocks. In ocamltest, all the values of variables
|
||||
are strings. Here are a few examples of things that can be achieved just
|
||||
by defining the appropriate variables. The complete description of the
|
||||
actions provided by ocamltest and the variables they use will be given
|
||||
in chapters [[#builtins]] and [[#ocaml-specific]].
|
||||
|
||||
*** Passing flags to the compilers
|
||||
|
||||
Assume our =hello.ml= example is modified as follows:
|
||||
|
||||
#+begin_src
|
||||
(* TEST *)
|
||||
|
||||
open Format
|
||||
|
||||
let _ = print_endline "Hello, world!"
|
||||
#+end_src
|
||||
|
||||
As may be verified, this program still passes the default tests. It is
|
||||
however not as minimal as our previous version, because the =Format=
|
||||
module is opened but not used. Fortunately, OCaml has a warning to
|
||||
detect such unused =open= directives, namely warning 33, which is
|
||||
disabled by default. We could thus add this version of =hello.ml=
|
||||
to the test suite, not so much to verify that the program compiles and
|
||||
runs as expected (we verified this already), but rather to make sure
|
||||
the compiler does indeed trigger the expected warning. Here are the
|
||||
required steps to achieve this:
|
||||
|
||||
1. We slightly modify the test block in =hello.ml=, as follows:
|
||||
#+begin_src
|
||||
(* TEST
|
||||
flags = "-w +33"
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
2. Since we now expect a non-empty output for the compilers, we need to
|
||||
store the expected output in a file, namely =hello.compilers.output=
|
||||
besides to =hello.ml= and =hello.reference=. To figure out what
|
||||
this file shall contain, we can run ocamltest even before it
|
||||
has been created. Of course, the action that checks compiler output
|
||||
will fail, but in this way we will get the compiler's output
|
||||
which we will just have to check (to
|
||||
make sure it is what we expect) and to move to the reference file.
|
||||
Thus, we do:
|
||||
: $ ocamltest hello.ml
|
||||
which fails, unsurprisingly, and shows us the paths to the file
|
||||
containing the output produced by the compiler and the path to the
|
||||
expected reference file. We also see what the compiler produced as
|
||||
output but we can double-check that the output is what we expect as a
|
||||
reference:
|
||||
: $ cat _ocamltest/hello/ocamlc.byte/ocamlc.byte.output
|
||||
which shows the warning we expect from the compiler. We can thus move
|
||||
this file to the reference file:
|
||||
: $ mv _ocamltest/hello/ocamlc.byte/ocamlc.byte.output hello.compilers.reference
|
||||
and if we now run ocamltest again, all the tests pass.
|
||||
|
||||
Two remarks are due. First, we have used the =flags= variable, to pass
|
||||
extra flags to all the compilers. There are two other variables one can
|
||||
use, namely =ocamlc_flags= and =ocamlopt_flags=, to pass flags to the
|
||||
bytecode or native compilers. Second, in this test all the compilers
|
||||
have the same output so one reference file is enough for all of them.
|
||||
There are situations, though, where the compiler's output is
|
||||
back-end-specific (it depends whether we compile to bytecode or to native
|
||||
code) or even compiler-specific. ocamltest is clever enough to know how
|
||||
to deal with such situations, provided that the reference files are
|
||||
named appropriately. It will indeed first lookup the test source
|
||||
directory for a compiler-specific reference file, e.g.
|
||||
=hello.ocamlc.byte.reference=. If no such file exists, a
|
||||
back-end-specific reference file is searched, e.g.
|
||||
=hello.ocamlc.reference= for a compiler common to both =ocamlc.byte= and
|
||||
=ocamlc.opt=. If this file does not exist either, ocamltest falls back
|
||||
to looking for =hello.compilers.reference= as we have seen in this
|
||||
example, the absence of which meaning that the compiler's output is
|
||||
expected to be empty.
|
||||
|
||||
*** Using an auxiliary module
|
||||
|
||||
Let's start with our original =hello.ml= test program and extract the
|
||||
greeting logic into a distinct =greet.ml= module:
|
||||
|
||||
#+begin_src
|
||||
let greet guest = Printf.printf "Hello, %s!\n" guest
|
||||
#+end_src
|
||||
|
||||
Let's also write an interface, =greet.mli=:
|
||||
|
||||
#+begin_src
|
||||
val greet : string -> unit
|
||||
#+end_src
|
||||
|
||||
Our =hello.ml= test program can then be rewritten as follows:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
modules = "greet.ml"
|
||||
*)
|
||||
|
||||
let _ = Greet.greet "world"
|
||||
#+end_src
|
||||
|
||||
Provided that the =hello.compilers.reference= file previously to test
|
||||
warnings is deleted, running ocamltest on =hello.ml= should work. It
|
||||
will also be worth looking at the two first lines of the log file generated
|
||||
while running the test. It says:
|
||||
|
||||
#+begin_src
|
||||
Specified modules: greet.ml hello.ml
|
||||
Source modules: greet.mli greet.ml hello.ml
|
||||
#+end_src
|
||||
|
||||
The first line shows that the =modules= variable has been taken into
|
||||
account. On the second line, it can be seen that the =greet.mli= file
|
||||
appears, right before =greet.ml=. It is ocamltest that has added it,
|
||||
because it has been recognized as an interface for one of the specified
|
||||
modules.
|
||||
|
||||
To sum up, if a test consists in several modules, it is enough to list
|
||||
their implementations (except the one of the main test program which is
|
||||
implicit) in the =modules= variable, in linking order. There is no need
|
||||
to worry about their interfaces, which will be added automatically by
|
||||
ocamltest, if they exist.
|
||||
|
||||
*** Linking with a library
|
||||
|
||||
Assume we want to use the following program to make sure regular
|
||||
expressions as implemented by the =Str= library work as expected:
|
||||
|
||||
#+begin_src
|
||||
let hello_re = Str.regexp "^Hello, .+!$"
|
||||
|
||||
let hello_str = "Hello, world!"
|
||||
|
||||
let _ =
|
||||
if not (Str.string_match hello_re hello_str 0) then
|
||||
begin
|
||||
Printf.eprintf "There is a problem!\n";
|
||||
exit 2
|
||||
end
|
||||
#+end_src
|
||||
|
||||
This test terminates silently if everything goes well and prints a
|
||||
message on its standard error only if something goes wrong, which means
|
||||
we won't have anything special to do so that ocamltest checks for an
|
||||
empty output after the program has run. However, to be able to compile
|
||||
and link this test, there are several things we need to do so that it
|
||||
finds the =Str= library it uses. More precisely, we need to add the =-I=
|
||||
option pointing to the right directory and, at link time, to give the
|
||||
name of the appropriate library file. To make our life a bit simpler,
|
||||
ocamltest has a few variables where directories and libraries can be
|
||||
listed. Once they are there, it is ocamltest which will take care of
|
||||
adding the =-I= option for each directory and for adding the right
|
||||
library file depending on whether we are producing bytecode or native
|
||||
programs. So, here is how the previous program can be annotated so that
|
||||
it becomes a test:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
directories += " ${ocamlsrcdir}/otherlibs/str "
|
||||
libraries += " str "
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
With these annotations, it becomes possible to run =re.ml= as an
|
||||
ocamltest test program and, doing so, one may notice that the two tests
|
||||
pass. There are however a few other things worth pointing out here
|
||||
regarding the ocamltest DSL. For a start, the notation =${variable}=
|
||||
inside a string means to replace =variable= by its value, as happens in
|
||||
many other languages, like the bash shell. Moreover, it is the first
|
||||
time we meet the ~+=~ operator which concatenates a value to a variable.
|
||||
More precisely,
|
||||
: foo += "bar"
|
||||
is equivalent to
|
||||
: foo = "${foo}bar"
|
||||
and not to
|
||||
: foo = "${foo} bar"
|
||||
as it may happen in other languages such as makefiles.
|
||||
|
||||
In other words, the ~+=~ operator concatenates two strings without
|
||||
inserting any implicit space between them as e.g. make would do. This is
|
||||
because in some cases such a behavior is required and could not be
|
||||
achieved if spaces were implicitly added, whereas with a literal
|
||||
concatenation it is always possible to include spaces explicitly. This is
|
||||
exactly what happens in the ocamltest annotation block above, where the
|
||||
strings added to the =libraries= and =directories= variables are
|
||||
surrounded by spaces. As should be clear to the reader by now, these
|
||||
spaces are mandatory. Without them, the added values would be glued to
|
||||
the last word of the variable and would thus be misinterpreted.
|
||||
|
||||
Finally, one may notice that, although ocamltest does make it
|
||||
possible to link a test program with a library, it does not really make
|
||||
it easy or convenient to do so. In particular, what if we want to write
|
||||
several, perhaps many test programs that need to be linked with =Str=?
|
||||
Will we have to repeat these lines everywhere, thus creating code that
|
||||
is going to be tedious to maintain? Well, fortunately not. Actually,
|
||||
ocamltest has a much more elegant way to deal with such issues, namely
|
||||
/environment modifiers/. As will be explained in chapter [[#concepts]], an
|
||||
/environment modifier/ is an object that gathers several variable
|
||||
definitions that can then be included in an ocamltest block at once.
|
||||
Environment modifiers have to be defined in ocamltest itself and can
|
||||
then be used with the =include= directive. For instance, the previous
|
||||
test block is actually written as follows:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
include str
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
*** Testing only on Unix systems
|
||||
|
||||
So far, we have been able to fulfill our requirements just by assigning
|
||||
the right values to variables and relying on the =bytecode= and =native=
|
||||
tests ocaml runs by default. There are however situations where this is
|
||||
not enough and where one needs the ability to run other tests. One
|
||||
example of such a situation is when a test needs to be performed only on
|
||||
one given operating system, e.g. because it uses a feature which is
|
||||
present only on that operating system. On an other operating system, the
|
||||
test should be skipped because it is irrelevant. To illustrate this,
|
||||
here is how our original =hello.ml= test program should be annotated so
|
||||
that it is run only on Unix platforms:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
:* unix
|
||||
:** bytecode
|
||||
:** native
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
As can be understood from this example, lines starting with an asterisk
|
||||
describe which tests should be executed. In addition, the number of
|
||||
asterisks allows to specify the nesting level of each test or action.
|
||||
Here for instance, =bytecode= and =native= are sub-tests that will be
|
||||
run only if the =unix= test passes and will not be started if it fails
|
||||
or skips.
|
||||
|
||||
This way of describing the dependencies between tests has been inspired
|
||||
by the syntax of org-mode. Each line starting with asterisks (thus lines
|
||||
specifying which tests to run) can also be seen as a title. The whole
|
||||
set of lines is like the outline of the test scenario.
|
||||
|
||||
With this information in mind, it can be seen that the smallest test
|
||||
block
|
||||
: (* TEST *)
|
||||
is actually equivalent to
|
||||
#+begin_src
|
||||
(* TEST
|
||||
:* bytecode
|
||||
:* native
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
One common error when designing tests is to believe that a block like
|
||||
#+begin_src
|
||||
(* TEST
|
||||
:* unix
|
||||
*)
|
||||
#+end_src
|
||||
means to execute the =unix= test that verifies that the OS is indeed
|
||||
Unix and then to execute the default tests. This is actually not the
|
||||
case. The only situation in which the default tests are considered is
|
||||
when the test block contains absolutely no line starting with an
|
||||
asterisk. As soon as there is a line starting with an asterisk, the
|
||||
default tests are ignored completely and one needs to be totally
|
||||
explicit about which tests to run. So the correct way to write the
|
||||
erroneous block above is the use shown at the beginning of this section,
|
||||
namely:
|
||||
#+begin_src
|
||||
(* TEST
|
||||
:* unix
|
||||
:** bytecode
|
||||
:** native
|
||||
*)
|
||||
#+end_src
|
||||
|
||||
The fact that the language is inspired by org-mode should also be
|
||||
helpful in understanding the scope of variable assignments. Roughly
|
||||
speaking:
|
||||
|
||||
1. Variables defined at the root level are visible by all the tests and
|
||||
sub-tests that follow their assignment.
|
||||
|
||||
2. If a variable is defined just below a test line, then it is visible
|
||||
by that test and all its sub-tests (unless its definition is
|
||||
overridden) but not by tests at a nesting level whose depth is less or
|
||||
equal than the one of the test in which the variable is defined.
|
||||
|
||||
For instance, given the following block:
|
||||
#+begin_src
|
||||
(* TEST
|
||||
foo = "abc"
|
||||
:* test1
|
||||
bar = "def"
|
||||
:** subtest1
|
||||
baz = "hij"
|
||||
:** subtest2
|
||||
:* test2
|
||||
*)
|
||||
#+end_src
|
||||
- The definition of =foo= is visible in all the tests
|
||||
|
||||
- The definition of =bar= is visible in all the tests except =test2=.
|
||||
|
||||
- The definition of =baz= is visible only in =subtest1=.
|
||||
|
||||
** Other useful tests
|
||||
|
||||
This section introduces three tests provided by =ocamltest= and that can
|
||||
be of particular interest. A complete list of available tests and
|
||||
actions and their detailed descriptions are given in chapters
|
||||
[[#builtins]] and [[#ocaml-specific]].
|
||||
|
||||
*** Testing the top-level: the =toplevel= and =expect= tests
|
||||
|
||||
Two tests are provided to make sure that the OCaml top-level behaves as
|
||||
expected: =toplevel= and =expect=. These tests are similar in that they
|
||||
both allow to test how the OCaml top-level reacts to some user input,
|
||||
but they are different in the way one specifies the expected output and
|
||||
also in what they can test. The =toplevel= test behaves in a spirit
|
||||
similar to the compiler tests described above, meaning that the expected
|
||||
output has to be stored in its own, separate file. Since this test
|
||||
invokes the real OCaml top-level, it is useful to test advanced features
|
||||
like the behavior of the top-level when its input is a file rather than
|
||||
a terminal, or similar things. In the expect test, on the contrary,
|
||||
the input and the output it is expected to produce can be written in
|
||||
the same file, close to each other. However, this test uses the OCaml
|
||||
top-level as a library, rather than calling it as an external program.
|
||||
So this test is actually not testing the complete real OCaml top-level,
|
||||
but for testing language features it remains perfectly valid and is
|
||||
actually what is needed in most of the cases. We thus give below an
|
||||
example of an expect test and will describe the =toplevel= test in
|
||||
chapter [[#ocaml-specific]].
|
||||
|
||||
So, here is a toy example of an =expect= test:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
:* expect
|
||||
*)
|
||||
|
||||
type point = { x : int; y : int };;
|
||||
[%%expect{|
|
||||
type point = { x : int; y : int; }
|
||||
|}];;
|
||||
#+end_src
|
||||
|
||||
The first line after the test block is the input phrase, while the line
|
||||
that appears between =[%%expect{|= and =|}];;= is the corresponding
|
||||
expected output. The =expect= test can also be used to test the output
|
||||
in presence of the =-principal= command-line flag. In such cases, the
|
||||
expected output should be written in a =|}, Principal{|= block (to be
|
||||
improved).
|
||||
|
||||
*** The =script= test
|
||||
|
||||
It may happen that a needed test is not provided by ocamltest. Of
|
||||
course, if it turns out that this test would be helpful to test several
|
||||
source files, then the best solution is to add it to ocamltest itself.
|
||||
Some tests are however so specific that it is easier to write them as
|
||||
shell scripts. Such tests can be run by the =script= test, their name
|
||||
being defined by the =script= variable. In this case, the script is run
|
||||
in an environment where all the variables defined in ocamltest have been
|
||||
exported. The script uses its exit status to report its result and can
|
||||
write a response to a dedicated file to modify its environment or
|
||||
explain why it failed or skipped, as will be explained in chapter
|
||||
[[#builtins]]. For the moment, let's see how to use a script to "test" our
|
||||
original =hello.ml= example. Our annotated program would look as
|
||||
follows:
|
||||
|
||||
#+begin_src
|
||||
(* TEST
|
||||
script = "${test_source_directory}/faketest.sh"
|
||||
:* script
|
||||
*)
|
||||
|
||||
let _ = print_endline "Hello, world!"
|
||||
#+end_src
|
||||
|
||||
And here is =faketest.sh=, make sure it is executable:
|
||||
|
||||
#+begin_src
|
||||
#!/bin/sh
|
||||
exit ${TEST_PASS}
|
||||
#+end_src
|
||||
|
||||
This should be enough for the following command to work:
|
||||
: ocamltest hello.ml
|
||||
|
||||
This of course tests nothing and a real test script should actually do
|
||||
something before returning its result. Let's however see how we can
|
||||
make the script test fail gracefully:
|
||||
|
||||
#+begin_src
|
||||
#!/bin/sh
|
||||
echo Why should this pass in the first place > ${ocamltest_response}
|
||||
exit ${TEST_FAIL}
|
||||
#+end_src
|
||||
|
||||
Running ocamltest on our =hello.ml= program again produces the following
|
||||
output:
|
||||
#+begin_src
|
||||
... testing 'hello.ml' with 1 (script) => failed (Why should this pass in the first place)
|
||||
#+end_src
|
||||
|
||||
* Key concepts
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: concepts
|
||||
:END:
|
||||
|
||||
** Actions, hooks and tests
|
||||
|
||||
** Semantics of a test block
|
||||
|
||||
** Variables, environments and how they are inherited
|
||||
|
||||
** Environment modifiers
|
||||
|
||||
* Built-in actions and tests
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: builtins
|
||||
:END:
|
||||
|
||||
* OCaml-specific actions and tests
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: ocaml-specific
|
||||
:END:
|
||||
|
||||
# Things to document (requested by Leo on caml-devel)
|
||||
# - the syntax of the DSL
|
||||
# - the precise meaning of the stars
|
||||
# - a clear definition of what "test" means in the context of the DSL
|
||||
# - a list of the builtin "actions"
|
||||
# - a list of which "actions" depend on which "variables"
|
||||
# - what does "include" do?
|
||||
# - what is the scoping of variables?
|
||||
|
||||
# LocalWords: ocamltest OCaml DSL extensibility makefiles
|
||||
|
||||
# Local Variables:
|
||||
# ispell-local-dictionary: "english"
|
||||
# End:
|
||||
Loading…
Reference in New Issue