[Theora-dev] libtheora test suite

[Conrad Parker]

Hi,

I just committed the framework for a test suite, and a couple of small
tests, to libtheora (changeset:9700).

The tests live in a directory called tests. When building with GNU autotools:

  * 'make check' will run the test programs listed in the TESTS variable in
  tests/Makefile.am . 'make check' will fail if any tests fail.

  * 'make distcheck'  will fail if any tests fail. 'make distcheck' should
  be used to create distribution tarballs. The point of that is that you
  ensure all regression tests pass before release. ('make distcheck' has
  other nice benefits like testing that install/uninstall works correctly).

  * './configure --enable-valgrind-testing' will prepare things so that
  'make check' will run tests under valgrind in order to detect memory leaks.

The tests I added are fairly trivial. One is a 'noop' test which simply
creates and destroys each kind of data structure the library provides.
This is useful as a baseline test, and to check (via valgrind) whether the
library contains memory leaks in its constructors and destructors.

This is the output of 'make check' as at changeset:9700:

make  check-TESTS
make[2]: Entering directory `/home/conrad/src/xiph.org/theora/tests'
----  + Initializing theora_info struct ...
----  + Initializing theora_state for encoding ...
----  + Clearing theora_state ...
----  + Clearing theora_info struct ...
==9720== 
==9720== 256 bytes in 1 blocks are definitely lost in loss record 1 of 1
==9720==    at 0x1B90659D: malloc (vg_replace_malloc.c:130)
==9720==    by 0x1B9357E4: oggpack_writeinit (bitwise.c:41)
==9720==    by 0x1B93582E: oggpackB_writeinit (bitwise.c:47)
==9720==    by 0x1B9213CE: theora_encode_init (encoder_toplevel.c:845)
==9720==    by 0x8048735: noop_test_encode (noop.c:15)
==9720==    by 0x804886D: main (noop.c:62)
PASS: noop
----  + Initializing theora_comment ...
----  + Adding ARTIST1 ...
----  + Adding LICENSE by tag ...
----  + Adding ARTIST2 by tag ...
----  + Querying value of LICENSE ...
----  + Querying count of ARTIST comments ...
----  + Querying value of ARTIST index 0 ...
----  + Querying value of ARTIST index 1 ...
----  + Querying value of ARTIST index 2 (out of bounds) ...
----  + Querying value of UNDEF index 7 (tag not defined) ...
----  + Clearing theora_comment ...
PASS: comment-test
==================
All 2 tests passed
==================

The logic of the tests passed, but valgrind shows a leak in the encoder.

This leak is fixed in a one-liner in changeset:9701 [and the crowd goes
wild!! valgrind wins again!!]

Next we see the verbose output of the comment-test. This test uses all of
the theora_comment_*() API functions, and checks the correctness of return
values and errors. If we're happy that the test covers all comment functions,
then we can be reasonably happy that if it passes, the comments API is:

	* completely exported by the linker (the test runs at all),
	* does not contain any memory leaks (as valgrind doesn't complain),
	* and is correctly implemented (as the test passes).

Ideally, we'd write tests to cover all the encoder and decoder functionality
too.

So, the point of this email is:

	1) to inform about the new test suite framework in libtheora,
	2) to encourage use of 'make check' and 'make distcheck',
	3) to encourage people to write more tests to cover the API, and
	4) to share the instant gratification of a fixing a one-line leak.

If you haven't rolled your eyes too much by this point you will have realized
that the noop test above didn't test the decoder. I've got some questions
about that so I'll leave it for a separate email :-)

cheers,

Conrad.