Next: Running regress.pike, Previous: Test Suites, Up: Regression
./test.sh blunder.tst
runs the tests in blunder.tst and
prints the results of the commands on numbered lines, which may look
like:
1 E5 2 F9 3 O18 4 B7 5 A4 6 E4 7 E3 8 A3 9 D9 10 J9 11 B3 12 C6 13 C6
This is usually not very informative, however. More interesting is
./eval.sh blunder.tst
which also compares the results above
against the correct ones in the test file and prints a report for each
test on the form:
1 failed: Correct '!E5', got 'E5' 2 failed: Correct 'C9|H9', got 'F9' 3 PASSED 4 failed: Correct 'B5|C5|C4|D4|E4|E3|F3', got 'B7' 5 PASSED 6 failed: Correct 'D4', got 'E4' 7 PASSED 8 failed: Correct 'B4', got 'A3' 9 failed: Correct 'G8|G9|H8', got 'D9' 10 failed: Correct 'G9|F9|C7', got 'J9' 11 failed: Correct 'D4|E4|E5|F4|C6', got 'B3' 12 failed: Correct 'D4', got 'C6' 13 failed: Correct 'D4|E4|E5|F4', got 'C6'
The result of a test can be one of four different cases:
passed
: An expected pass
This is the ideal result.
PASSED
: An unexpected pass
This is a result that we are hoping for when we fix a bug. An old test case that used to fail is now passing.
failed
: An expected failure
The test failed but this was also what we expected, unless we were trying to fix the particular mistake highlighted by the test case. These tests show weaknesses of the GNU Go engine and are good places to search if you want to detect an area which needs improvement.
FAILED
: An unexpected failure
This should nominally only happen if something is broken by a change. However, sometimes GNU Go passes a test, but for the wrong reason or for a combination of wrong reasons. When one of these reasons is fixed, the other one may shine through so that the test suddenly fails. When a test case unexpectedly fails, it is necessary to make a closer examination in order to determine whether a change has broken something.
If you want a less verbose report, ./regress.sh . blunder.tst
does the same thing as the previous command, but only reports unexpected
results. The example above is compressed to
3 unexpected PASS! 5 unexpected PASS! 7 unexpected PASS!
For convenience the tests are also available as makefile targets. For
example, make blunder
runs the tests in the blunder test suite by
executing eval.sh blunder.tst
. make all_batches
runs all
test suites in a sequence using the regress.sh
script.