irrlicht/tests/tests-readme.txt

Welcome to the Irrlicht test suite.
===================================
This is composed of a series of tests which exercise basic Irrlicht functionality.  These are not
strictly unit tests, since there is no stub framework that isolates each method under test.  They
do however test small units of functionality and should help to isolate problems and spot regressions.

You are encouraged to run the tests whenever you make any significant code change, and to add tests
for any new or modified area of code.

The overall test application will return a count of the number of tests that failed, i.e. 0 is success.
It will also log to tests/tests.log file, and on success will update the tests/tests-last-passed-at.txt
file (which gets committed with code changes as a very basic verification that we're not regressing).


Working directory
=================
Since the tests rely on the presence of /media and /empty/empty subdirectories, the working directory
must be the /tests directory, not /bin/$PLATFORM. This means that you cannot run /bin/$PLATFORM/texts.exe
from there.  You can however cd to /tests and run ../bin/$PLATFORM/tests.exe


Adding a new test
=================
To add a new test, e.g. "myNewTest":

1) Create tests/myNewTest.cpp.  At a minimum, this must contain a function with the signature
   bool fnName(void), where fnName is the same as the filename (without the .cpp suffix).

   This function must return true if the tests passes, or false if it fails. In this example,
   the function should be:

   bool myNewTest(void)
   {
   ...
   }

2) Add myNewTest.cpp to the build targets in tests.cbp, tests_vc8.vcproj and tests_vc9.vcproj.  These
   are all text files that can be edited manually by hand; just copy, paste and modify an existing
   source file entry.

3) In tests/main.cpp, find the list of TEST() macro calls, and add a call to your new test, using the
   TEST macro, e.g.:

   TEST(myNewTest);

4) Run your test, and verify any images that it produces (see "Screenshots").

5) Remember to svn add tests/newNewTest.cpp and any new tests/media/ files.

Your test will be run independently in its own indepedent process. It is responsible for creating any
required resources or Irrlicht interfaces, and for cleaning up after itself and restoring the working
directory to /tests.  You do not have to link against Irrlicht.lib; the whole application is already
linked to it.


Logging
=======
Please use logTestString() to log any intersting output or fails from your test.  This is declared in
tests/testUtils.h.  Its output goes to tests/tests.log


Screenshots
===========
testUtils.h/.cpp provides a function to create a screenshot and compare it with a reference image.
This is useful for validating new or changed functionality, and for catching regressions.

Call the unambiguously named takeScreenshotAndCompareAgainstReference() function to do this. It needs
an IVideoDriver (which it will use to create the first part of the filename) and a unique filename
including an image format suffix, e.g. "-myNewTest.png".  You should use .png as a suffix unless you
have a very specific need to use another format.  Please avoid using .jpg as image compression can
introduce artifacts and false fails.

Optionally, you can specify the amount of match that is required between the produced screenshot and
the reference image.  While the images should match exactly, we have found that OpenGL implementations
can vary significantly across test machines, so a default 99% match (of total colour values across all
pixels) is assumed.  You may have to go as low as 98% for some images, but please try to err on the side
of strictness until we can determine that your test image needs to be fuzzier on other peoples' machines.

If takeScreenshotAndCompareAgainstReference() can't find an existing reference image, it will create
one from the screenshot.  In effect, this means that you have to run your test once (expecting it to
fail) in order to produce the initial reference images.  The new images are created in tests/results
with filename:

driverName-filename.suffix

e.g. OpenGL-myNewTest.png (note that the OpenGL driver elides the actual OpenGL version number from
the filename, as this tends to differ between machines and installations).

You should check these new images carefully to ensure that they show exactly what you expect.  Please
do not just assume that they do, as validating bad behaviour is worse than not validating it at all!

If the images do appear exactly as you expect, move them to the tests/media directory, and re-run the
tests.  They should now pass.  Remember to svn add any new media files!


What to do when the tests fail
==============================
DON'T PANIC!

This is a Good Thing.  Failing tests challenge our assumptions and help us to make Irrlicht more robust.

First, check your working directory.  The tests need to be run from the tests/ directory, not a /bin
subdirectory.  You can do this using the working directory in your debugger, or on the command line by
running the tests executable (wherever it is build) from the tests/ directory.

If you need to debug a test, first move it temporarily to the start of the list of TEST() macros. This
is because each test runs in its own process, so only the first test will have the debugger attached.

If the fail is due to a bitmap difference, carefully compare the bitmaps, and the amount of failure. The
OpenGL device does tend to produce differences.  You should not just automatically make a test fuzzier,
but if you can rule out any real issue in the code, it can be valid to accept OpenGL image matches as low
as 98%.  Other devices should not require this amount of fuzziness!

If you can't figure out the reason for the failure (or better yet, if you can, and think the tests and/or
Irrlicht need updated), then please do raise the issue in the bug forum:

http://irrlicht.sourceforge.net/phpBB2/viewforum.php?f=7

We do want to hear about fails, and will thank you for finding them.