From 1c0424c80efe3f7b6a4aa838de6ba78c1d9fa607 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Hinderer?= Date: Thu, 28 Nov 2019 17:35:53 +0100 Subject: [PATCH] ocamltest: draft of reference manual (tutorial section) --- .gitignore | 1 + Changes | 3 + ocamltest/Makefile | 10 + ocamltest/ocamltest.org | 745 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 759 insertions(+) create mode 100644 ocamltest/ocamltest.org diff --git a/.gitignore b/.gitignore index 5da73a826..b9c3eeca2 100644 --- a/.gitignore +++ b/.gitignore @@ -120,6 +120,7 @@ _build /ocamltest/tsl_lexer.ml /ocamltest/tsl_parser.ml /ocamltest/tsl_parser.mli +/ocamltest/ocamltest.html /otherlibs/dynlink/extract_crc /otherlibs/dynlink/dynlink_platform_intf.mli diff --git a/Changes b/Changes index d6a97602c..779f16e46 100644 --- a/Changes +++ b/Changes @@ -33,6 +33,9 @@ Working version ### Manual and documentation: +- #9141: beginning of the ocamltest reference manual + (Sébastien Hinderer, review by Gabriel Scherer and Thomas Refis) + ### Compiler user-interface and warnings: - #9074: reworded error message for non-regular structural types diff --git a/ocamltest/Makefile b/ocamltest/Makefile index 59ca8a624..bcb842120 100644 --- a/ocamltest/Makefile +++ b/ocamltest/Makefile @@ -261,6 +261,15 @@ ocamltest_config.ml: ocamltest_config.ml.in Makefile ../Makefile.config -e 's|@@FUNCTION_SECTIONS@@|$(FUNCTION_SECTIONS)|' \ $< > $@ +# Manual + +.PHONY: doc + +doc: ocamltest.html + +ocamltest.html: ocamltest.org + pandoc -s --toc -N -f org -t html -o $@ $< + .PHONY: clean clean: rm -rf ocamltest$(EXE) ocamltest.opt$(EXE) @@ -269,6 +278,7 @@ clean: rm -rf $(cmo_files) rm -rf $(cmx_files) rm -rf $(generated) + rm -f ocamltest.html ifneq "$(TOOLCHAIN)" "msvc" .PHONY: depend diff --git a/ocamltest/ocamltest.org b/ocamltest/ocamltest.org new file mode 100644 index 000000000..20651fc70 --- /dev/null +++ b/ocamltest/ocamltest.org @@ -0,0 +1,745 @@ +#+STARTUP: showall + +#+title: The ocamltest reference manual +#+language: en + +#+HTML_HEAD: + +* Introduction + +This is =verbatim= and this is ~code~. + +** What is ocamltest + +ocamltest is a test-driver, that is, a program that can run tests and report +their results so that they can be used by a test infrastructure. + +Originally, the tool has been designed specifically to run the integration +tests of the OCaml compiler's test suite. However, it has been +designed with extensibility in mind and thus has a plugin +mechanism that makes it possible to extend it with other tests. + +** Design choices + +*** Programming language and external dependencies + +For a start, one may wonder in which language a test-driver for a compiler +should be written. It may indeed seem odd to write a test-driver for a +compiler in the language it compiles, since the compiler itself +is yet untested and thus not trustworthy. + +It can however be observed that the OCaml compiler is /bootstraped/, +meaning that it is itself written in OCaml. A newer version of the +compiler can thus be produced from an existing one and the (OCaml) +source code of that newer version. Practically, this means that the +compiler works at least well enough to recompile itself. This is why we +consider that it is okay to write a test-driver like ocamltest in OCaml, +as long as it uses only code that has been used to bootstrap the +compiler. In particular, this is why we prefer not to rely on any +external dependency, not even the libraries included in the compiler +distribution such as the =Unix= library. + +*** Test types + +As has been noted above, ocamltest has been developed to run the already +existing integration tests of the OCaml compiler's test suite, which +were previously run by a set of makefiles. This context explains +several design decisions which could otherwise seem rather arbitrary. + +For example, the reason why ocamltest has no support for running unit tests +is that there were no such tests in the OCaml compiler's test suite. + +Indeed, the OCaml compiler's test suite is composed mainly of complete +programs. In this context, the most current meaning of "testing" a program +is that the program needs to be compiled and executed. The test will +be considered successful if the program compiles as expected and, when run, +returns the expected value. + +Since this scenario is the most frequent one, it was of particular +importance to make writing tests of this form as simple as possible. + +However, not all tests fall into the previously described category, so it is +also necessary to support not only variations on the previous scenario +(compile but do not run, compile with certain options, etc.) but also +completely different tests, such as top-level tests, debugger tests, +etc. + +To fulfill these requirements and make it as easy as possible to turn a +program into a test, it has been chosen to design a Domain-Specific +Language (DSL) used to annotate the test program with a +=(* TEST *)= block at its top. This block specifies how the test +should be performed. + +** Outline of this document + +The next chapter explains through examples how to write simple tests. We +then introduce the key concepts used by ocamltest to provide a better +understanding of how it works and can be used to write more complex +tests. The two last chapters give an in-depth description of the +built-in tests and actions and of the tests and actions that are specific +to the OCaml compiler. + +* Writing simple tests + +This chapter is a tutorial. It explains how to write simple test +programs and also tries to give insights about how ocamltest works. These +insights will be deepened in chapter [[#concepts]] where ocamltest is +presented in a more abstract and conceptual way. + +We start by explaining how to set-up a proper environment for writing +tests. We then show how to turn the traditional "Hello, world!" program +into a test and explain how to run it with ocamltest. We continue +with a few variations on this test and conclude this chapter +with a few other useful tests. + +** Prerequisites for writing tests + +Writing tests requires that the sources of the OCaml compiler for which +one wants to write them are downloaded and compiled. The compiler +does not need to be installed, though. + +The sources can be downloaded either as an archive, or directly cloned +through git, which seems more appropriate in the context of writing ones +own tests. Refer to +=INSTALL.adoc= (and also to =README.win32.adoc= if you are on Windows) to +learn how to get the sources of the OCaml compiler and to compile them. + +In the remainder of this manual, we will assume that the sources of the +OCaml compiler have been extracted in the =${OCAMLSRCDIR}= directory (for +instance =${HOME}/src/ocaml=) and that you have successfully configured +and compiled them as described in =INSTALL.adoc= or =README.win32.adoc=, +according to your operating system. The tools and libraries necessary +for running tests should also be built. This can be achieved by running +the following command from =${OCAMLSRCDIR}=: +: make -C testsuite lib tools + +We will also assume that an =ocamltest= command is available in +your =PATH=. Although this is not strictly necessary, it is strongly +recommended that you set this up because this will simplify test +development a lot. This can be achieved e.g. by creating a symbolic +link to =${OCAMLSRCDIR}/ocamltest/ocamltest= (or its native +counterpart =${OCAMLSRCDIR}/ocamltest/ocamltest.opt=) in a directory that +is already in your =PATH=, like =~/bin=. + +** Testing the "Hello, world!" program with the default tests + +*** Turning "Hello, world!" into a useful test program + +Consider the following OCaml implementation of the classical "Hello, world!" +program written to a =hello.ml= file: + +: let _ = print_endline "Hello, world!" + +Now assume we would like to make sure that the OCaml compiler can +compile this program and that the resulting executable indeed prints the +expected output. Here are the required steps to turn the program +above into a test usable by ocamltest to verify this: + +1. First, we add a special comment at the very beginning of our =hello.ml= + file to make it explicit that it is a test: + #+begin_src + (* TEST *) + + let _ = print_endline "Hello, world!" + #+end_src + +2. We then need to say what the expected outputs are. In our case, we + expect that compiling the test produces no output at all and that its + execution produces one single line: + : Hello, world! + To let ocamltest know about this, we create a =hello.reference= file + containing the program's expected output -- the line mentioned + above. There is nothing special to do for silent compilations + since this is what is expected by default and a non-silent + compilation would actually cause a test failure. + +3. We can now ask ocamltest to run our test program with the + following command: + : ocamltest hello.ml + + Running this would produce an output similar to this one: + + #+begin_src + ... testing 'hello.ml' with 1 (native) => passed + ... testing 'hello.ml' with 2 (bytecode) => passed + #+end_src + + In addition to this output, it may be noticed that the previous + command has also created an =_ocamltest= directory whose content will + be examined in the next sub-section. + +4. Finally, there is one extra step required if we want our newly created + test to be run automatically as part of the OCaml compiler's test suite. + We need to move =hello.ml= and =hello.reference= to a directory (say + =newtest=) located somewhere + below =testsuite/tests= in the compiler's source tree and we + need to declare the test. This is done by appending the name of the + file containing the =(* TEST *)= comment to an =ocamltests= + (mark the final s) file located in the =newtest= directory, + alongside the other files relevant to the test. Once this is done, + the command + : make all + executed in the =testsuite= directory of the OCaml compiler' source + tree will run all the test suite, which now also includes our own test. + +*** What exactly is going on during the test + +The only thing we know from ocamltest's output when run on =hello.ml= is +that it is running two tests named =bytecode= and =native= and that the two of +them succeed. This can seem rather uninformative, and in a way it is, but +it has to be kept in mind that this information is the one passed by the +test-driver (ocamltest) to the test infrastructure. In that respect, +this is enough. For us users, though, it is not. That's why +ocamltest logs much more details about what is going on in a per-test +log file, which should be located in the =_ocamltest/hello/hello.log= file +found in the directory where =hello.ml= is. + +Before looking at this log file, notice that it has been created in a +test-specific directory. ocamltest creates such a directory for each +file it tests and makes sure every file produced as a result of +testing this file will be placed in this directory, either directly, or +in one of its sub-directories. The latter happens if the test has +to be compiled several times, with the same compiler and different +command-line options, or with different compilers. In particular, +in order to better understand what follows, it may be helpful to +remember that =OCaml= actually consists in not less than four compilers: +=ocamlc.byte= and =ocamlc.opt= which are the bytecode and native +flavors of the bytecode compiler and =ocamlopt.byte= and +=ocamlopt.opt= which are the bytecode and native flavors of the native +compiler. So, as we will see, ''testing the bytecode compiler'' +actually involves testing two compilers, and the same goes for ''testing +the native compiler''. + +Now that all this has been spelled out, let's examine the log file +produced by the test. Although it is too long to be reproduced here, +it is recommended to go through it quickly to get an idea of its +structure. Here is how it starts: + +#+begin_src +Specified modules: hello.ml +Source modules: hello.ml +#+end_src + +The first line lists the names of the modules the test consists of. The +second line is almost similar but if some modules had separate +interface files, they would be listed here, too, without the user +having to specify them in the list of modules (for each specified =.ml= +file, ocamltest looks whether a corresponding =.mli= file exists and, if +so, adds it to the list of files to consider). + +The rest of the log file can be split into two parts which are very +similar to each other: one for the =native= test and one for the =bytecode= +test. Among other things, we learn that each of these tests is composed +of nine actions. Before diving into the details of what each of these +actions does, let us take this opportunity to introduce a bit of +ocamltest terminology. An /action/ is anything that can =pass=, =skip= or +=fail=. A =test= is a sequence of such actions. Running +a test thus means running each of its actions, in sequence, until all the +actions have been run or one of them returns =pass= or =skip=. Whatever +the last run action returns, this value will be the result of the whole +test. + +To give concrete examples of actions, let's briefly go over the nine ones +involved in the =bytecode= test (those for the =native= test are +quite similar): + +1. =setup-ocamlc.byte-build-env=:: as its name suggests, this action + creates a build environment where a program can be compiled and + executed using the =ocamlc.byte= compiler. More precisely, this + involves creating a dedicated directory under the test-file specific + directory and populating it with the files required by subsequent actions. + Depending on what the underlying operating system supports, the files + will be either symlinked or copied from the test source directory. + +2. =ocamlc.byte=:: invokes the =ocamlc.byte= compiler in various ways. + Here, the test program is compiled and linked, but as we will see + later, different behaviors are possible depending on ocamltest + /variables/. + +3. =check-ocamlc.byte-output=:: this action compares the compiler's + output to a reference file, if one exists. As has been mentioned + earlier, the absence of such a reference file specifies that the + compiler's output is expected to be empty -- if it is not, this + causes a failure of this action and thus of the whole =bytecode= + test. + +4. =run=:: now that the program has been successfully compiled, it is + run with its standard output and error streams saved to a file. + + +5. =check-program-output=:: this time it is the output of the program + which is compared to a reference file, namely the =hello.reference= + file created earlier. So far this comparison succeeds, because the + output of the program is identical to the reference file but, as an + exercise, one may try to modify the reference file to see how this + causes the failure of this action and of the whole =bytecode= test. + + This action concludes the test of the =ocamlc.byte= compiler. We now + know that it is able to successfully compile our test program and that + the resulting executable runs as expected. The four remaining actions + are going to test the =ocamlc.opt= compiler in a similar but not + identical way: + +6. =setup-ocamlc.opt-build-env=:: this action is the counterpart of + action 1 for the =ocamlc.opt= compiler. + +7. =ocamlc.opt=:: like action 2, this action compiles the test program + but with the =ocamlc.opt= compiler. + +8. =check-ocamlc.opt-output=:: again, this action is similar to + action 3. + +9. =compare-bytecode-programs=:: here we make sure that the generated + executable is correct, but in a different way than for the + =ocamlc.byte= compiler. Rather than running it and checking its + output, we compare it to the one produced in action 2. Such a check + may seem strange, because what it requires is that =ocamlc.byte= and + =ocamlc.opt= produce exactly the same binary and not two binaries + than perform similarly when they are run, but it has proven useful in + the past and has permitted to detect a subtle bug in the compiler. + +** Customizing the default tests + +As has been briefly mentioned, the precise behavior of actions (and +thus of tests) may depend on /variables/ whose value can be adjusted in +the =(* TEST ... *)= blocks. In ocamltest, all the values of variables +are strings. Here are a few examples of things that can be achieved just +by defining the appropriate variables. The complete description of the +actions provided by ocamltest and the variables they use will be given +in chapters [[#builtins]] and [[#ocaml-specific]]. + +*** Passing flags to the compilers + +Assume our =hello.ml= example is modified as follows: + +#+begin_src +(* TEST *) + +open Format + +let _ = print_endline "Hello, world!" +#+end_src + +As may be verified, this program still passes the default tests. It is +however not as minimal as our previous version, because the =Format= +module is opened but not used. Fortunately, OCaml has a warning to +detect such unused =open= directives, namely warning 33, which is +disabled by default. We could thus add this version of =hello.ml= +to the test suite, not so much to verify that the program compiles and +runs as expected (we verified this already), but rather to make sure +the compiler does indeed trigger the expected warning. Here are the +required steps to achieve this: + +1. We slightly modify the test block in =hello.ml=, as follows: + #+begin_src + (* TEST + flags = "-w +33" + *) + #+end_src + +2. Since we now expect a non-empty output for the compilers, we need to + store the expected output in a file, namely =hello.compilers.output= + besides to =hello.ml= and =hello.reference=. To figure out what + this file shall contain, we can run ocamltest even before it + has been created. Of course, the action that checks compiler output + will fail, but in this way we will get the compiler's output + which we will just have to check (to + make sure it is what we expect) and to move to the reference file. + Thus, we do: + : $ ocamltest hello.ml + which fails, unsurprisingly, and shows us the paths to the file + containing the output produced by the compiler and the path to the + expected reference file. We also see what the compiler produced as + output but we can double-check that the output is what we expect as a + reference: + : $ cat _ocamltest/hello/ocamlc.byte/ocamlc.byte.output + which shows the warning we expect from the compiler. We can thus move + this file to the reference file: + : $ mv _ocamltest/hello/ocamlc.byte/ocamlc.byte.output hello.compilers.reference + and if we now run ocamltest again, all the tests pass. + +Two remarks are due. First, we have used the =flags= variable, to pass +extra flags to all the compilers. There are two other variables one can +use, namely =ocamlc_flags= and =ocamlopt_flags=, to pass flags to the +bytecode or native compilers. Second, in this test all the compilers +have the same output so one reference file is enough for all of them. +There are situations, though, where the compiler's output is +back-end-specific (it depends whether we compile to bytecode or to native +code) or even compiler-specific. ocamltest is clever enough to know how +to deal with such situations, provided that the reference files are +named appropriately. It will indeed first lookup the test source +directory for a compiler-specific reference file, e.g. +=hello.ocamlc.byte.reference=. If no such file exists, a +back-end-specific reference file is searched, e.g. +=hello.ocamlc.reference= for a compiler common to both =ocamlc.byte= and +=ocamlc.opt=. If this file does not exist either, ocamltest falls back +to looking for =hello.compilers.reference= as we have seen in this +example, the absence of which meaning that the compiler's output is +expected to be empty. + +*** Using an auxiliary module + +Let's start with our original =hello.ml= test program and extract the +greeting logic into a distinct =greet.ml= module: + +#+begin_src +let greet guest = Printf.printf "Hello, %s!\n" guest +#+end_src + +Let's also write an interface, =greet.mli=: + +#+begin_src +val greet : string -> unit +#+end_src + +Our =hello.ml= test program can then be rewritten as follows: + +#+begin_src +(* TEST +modules = "greet.ml" +*) + +let _ = Greet.greet "world" +#+end_src + +Provided that the =hello.compilers.reference= file previously to test +warnings is deleted, running ocamltest on =hello.ml= should work. It +will also be worth looking at the two first lines of the log file generated +while running the test. It says: + +#+begin_src +Specified modules: greet.ml hello.ml +Source modules: greet.mli greet.ml hello.ml +#+end_src + +The first line shows that the =modules= variable has been taken into +account. On the second line, it can be seen that the =greet.mli= file +appears, right before =greet.ml=. It is ocamltest that has added it, +because it has been recognized as an interface for one of the specified +modules. + +To sum up, if a test consists in several modules, it is enough to list +their implementations (except the one of the main test program which is +implicit) in the =modules= variable, in linking order. There is no need +to worry about their interfaces, which will be added automatically by +ocamltest, if they exist. + +*** Linking with a library + +Assume we want to use the following program to make sure regular +expressions as implemented by the =Str= library work as expected: + +#+begin_src +let hello_re = Str.regexp "^Hello, .+!$" + +let hello_str = "Hello, world!" + +let _ = + if not (Str.string_match hello_re hello_str 0) then + begin + Printf.eprintf "There is a problem!\n"; + exit 2 + end +#+end_src + +This test terminates silently if everything goes well and prints a +message on its standard error only if something goes wrong, which means +we won't have anything special to do so that ocamltest checks for an +empty output after the program has run. However, to be able to compile +and link this test, there are several things we need to do so that it +finds the =Str= library it uses. More precisely, we need to add the =-I= +option pointing to the right directory and, at link time, to give the +name of the appropriate library file. To make our life a bit simpler, +ocamltest has a few variables where directories and libraries can be +listed. Once they are there, it is ocamltest which will take care of +adding the =-I= option for each directory and for adding the right +library file depending on whether we are producing bytecode or native +programs. So, here is how the previous program can be annotated so that +it becomes a test: + +#+begin_src +(* TEST +directories += " ${ocamlsrcdir}/otherlibs/str " +libraries += " str " +*) +#+end_src + +With these annotations, it becomes possible to run =re.ml= as an +ocamltest test program and, doing so, one may notice that the two tests +pass. There are however a few other things worth pointing out here +regarding the ocamltest DSL. For a start, the notation =${variable}= +inside a string means to replace =variable= by its value, as happens in +many other languages, like the bash shell. Moreover, it is the first +time we meet the ~+=~ operator which concatenates a value to a variable. +More precisely, +: foo += "bar" +is equivalent to +: foo = "${foo}bar" +and not to +: foo = "${foo} bar" +as it may happen in other languages such as makefiles. + +In other words, the ~+=~ operator concatenates two strings without +inserting any implicit space between them as e.g. make would do. This is +because in some cases such a behavior is required and could not be +achieved if spaces were implicitly added, whereas with a literal +concatenation it is always possible to include spaces explicitly. This is +exactly what happens in the ocamltest annotation block above, where the +strings added to the =libraries= and =directories= variables are +surrounded by spaces. As should be clear to the reader by now, these +spaces are mandatory. Without them, the added values would be glued to +the last word of the variable and would thus be misinterpreted. + +Finally, one may notice that, although ocamltest does make it +possible to link a test program with a library, it does not really make +it easy or convenient to do so. In particular, what if we want to write +several, perhaps many test programs that need to be linked with =Str=? +Will we have to repeat these lines everywhere, thus creating code that +is going to be tedious to maintain? Well, fortunately not. Actually, +ocamltest has a much more elegant way to deal with such issues, namely +/environment modifiers/. As will be explained in chapter [[#concepts]], an +/environment modifier/ is an object that gathers several variable +definitions that can then be included in an ocamltest block at once. +Environment modifiers have to be defined in ocamltest itself and can +then be used with the =include= directive. For instance, the previous +test block is actually written as follows: + +#+begin_src +(* TEST +include str +*) +#+end_src + +*** Testing only on Unix systems + +So far, we have been able to fulfill our requirements just by assigning +the right values to variables and relying on the =bytecode= and =native= +tests ocaml runs by default. There are however situations where this is +not enough and where one needs the ability to run other tests. One +example of such a situation is when a test needs to be performed only on +one given operating system, e.g. because it uses a feature which is +present only on that operating system. On an other operating system, the +test should be skipped because it is irrelevant. To illustrate this, +here is how our original =hello.ml= test program should be annotated so +that it is run only on Unix platforms: + +#+begin_src +(* TEST +:* unix +:** bytecode +:** native +*) +#+end_src + +As can be understood from this example, lines starting with an asterisk +describe which tests should be executed. In addition, the number of +asterisks allows to specify the nesting level of each test or action. +Here for instance, =bytecode= and =native= are sub-tests that will be +run only if the =unix= test passes and will not be started if it fails +or skips. + +This way of describing the dependencies between tests has been inspired +by the syntax of org-mode. Each line starting with asterisks (thus lines +specifying which tests to run) can also be seen as a title. The whole +set of lines is like the outline of the test scenario. + +With this information in mind, it can be seen that the smallest test +block +: (* TEST *) +is actually equivalent to +#+begin_src +(* TEST +:* bytecode +:* native +*) +#+end_src + +One common error when designing tests is to believe that a block like +#+begin_src +(* TEST +:* unix +*) +#+end_src +means to execute the =unix= test that verifies that the OS is indeed +Unix and then to execute the default tests. This is actually not the +case. The only situation in which the default tests are considered is +when the test block contains absolutely no line starting with an +asterisk. As soon as there is a line starting with an asterisk, the +default tests are ignored completely and one needs to be totally +explicit about which tests to run. So the correct way to write the +erroneous block above is the use shown at the beginning of this section, +namely: +#+begin_src +(* TEST +:* unix +:** bytecode +:** native +*) +#+end_src + +The fact that the language is inspired by org-mode should also be +helpful in understanding the scope of variable assignments. Roughly +speaking: + +1. Variables defined at the root level are visible by all the tests and + sub-tests that follow their assignment. + +2. If a variable is defined just below a test line, then it is visible + by that test and all its sub-tests (unless its definition is + overridden) but not by tests at a nesting level whose depth is less or + equal than the one of the test in which the variable is defined. + +For instance, given the following block: +#+begin_src +(* TEST +foo = "abc" +:* test1 +bar = "def" +:** subtest1 +baz = "hij" +:** subtest2 +:* test2 +*) +#+end_src +- The definition of =foo= is visible in all the tests + +- The definition of =bar= is visible in all the tests except =test2=. + +- The definition of =baz= is visible only in =subtest1=. + +** Other useful tests + +This section introduces three tests provided by =ocamltest= and that can +be of particular interest. A complete list of available tests and +actions and their detailed descriptions are given in chapters +[[#builtins]] and [[#ocaml-specific]]. + +*** Testing the top-level: the =toplevel= and =expect= tests + +Two tests are provided to make sure that the OCaml top-level behaves as +expected: =toplevel= and =expect=. These tests are similar in that they +both allow to test how the OCaml top-level reacts to some user input, +but they are different in the way one specifies the expected output and +also in what they can test. The =toplevel= test behaves in a spirit +similar to the compiler tests described above, meaning that the expected +output has to be stored in its own, separate file. Since this test +invokes the real OCaml top-level, it is useful to test advanced features +like the behavior of the top-level when its input is a file rather than +a terminal, or similar things. In the expect test, on the contrary, +the input and the output it is expected to produce can be written in +the same file, close to each other. However, this test uses the OCaml +top-level as a library, rather than calling it as an external program. +So this test is actually not testing the complete real OCaml top-level, +but for testing language features it remains perfectly valid and is +actually what is needed in most of the cases. We thus give below an +example of an expect test and will describe the =toplevel= test in +chapter [[#ocaml-specific]]. + +So, here is a toy example of an =expect= test: + +#+begin_src +(* TEST +:* expect +*) + +type point = { x : int; y : int };; +[%%expect{| +type point = { x : int; y : int; } +|}];; +#+end_src + +The first line after the test block is the input phrase, while the line +that appears between =[%%expect{|= and =|}];;= is the corresponding +expected output. The =expect= test can also be used to test the output +in presence of the =-principal= command-line flag. In such cases, the +expected output should be written in a =|}, Principal{|= block (to be +improved). + +*** The =script= test + +It may happen that a needed test is not provided by ocamltest. Of +course, if it turns out that this test would be helpful to test several +source files, then the best solution is to add it to ocamltest itself. +Some tests are however so specific that it is easier to write them as +shell scripts. Such tests can be run by the =script= test, their name +being defined by the =script= variable. In this case, the script is run +in an environment where all the variables defined in ocamltest have been +exported. The script uses its exit status to report its result and can +write a response to a dedicated file to modify its environment or +explain why it failed or skipped, as will be explained in chapter +[[#builtins]]. For the moment, let's see how to use a script to "test" our +original =hello.ml= example. Our annotated program would look as +follows: + +#+begin_src +(* TEST +script = "${test_source_directory}/faketest.sh" +:* script +*) + +let _ = print_endline "Hello, world!" +#+end_src + +And here is =faketest.sh=, make sure it is executable: + +#+begin_src +#!/bin/sh +exit ${TEST_PASS} +#+end_src + +This should be enough for the following command to work: +: ocamltest hello.ml + +This of course tests nothing and a real test script should actually do +something before returning its result. Let's however see how we can +make the script test fail gracefully: + +#+begin_src +#!/bin/sh +echo Why should this pass in the first place > ${ocamltest_response} +exit ${TEST_FAIL} +#+end_src + +Running ocamltest on our =hello.ml= program again produces the following +output: +#+begin_src + ... testing 'hello.ml' with 1 (script) => failed (Why should this pass in the first place) +#+end_src + +* Key concepts + :PROPERTIES: + :CUSTOM_ID: concepts + :END: + +** Actions, hooks and tests + +** Semantics of a test block + +** Variables, environments and how they are inherited + +** Environment modifiers + +* Built-in actions and tests + :PROPERTIES: + :CUSTOM_ID: builtins + :END: + +* OCaml-specific actions and tests + :PROPERTIES: + :CUSTOM_ID: ocaml-specific + :END: + +# Things to document (requested by Leo on caml-devel) +# - the syntax of the DSL +# - the precise meaning of the stars +# - a clear definition of what "test" means in the context of the DSL +# - a list of the builtin "actions" +# - a list of which "actions" depend on which "variables" +# - what does "include" do? +# - what is the scoping of variables? + +# LocalWords: ocamltest OCaml DSL extensibility makefiles + +# Local Variables: +# ispell-local-dictionary: "english" +# End: