[theora-dev] Request for input on the preparation of a subjective video quality comparative study

[Timothy B. Terriberry]

John Pisokas wrote:
> I would be happy to have your opinions, advice, views or anything else you
> wish.

I assume you are consulting documents such as ITU-R BT.500-11
"Methodology for the subjective assessment of the quality of television
pictures," which contain advice on setting up and administering the
tests, selection of test material, etc. ITU-T SG 12 COM 12-67 "VQEG
subjective test plan" may also be informative. DSCQS is generally
regarded as the most reliable evaluation method, and typically involves
short, 10s test clips. A wide variety of suitable clips are available at
http://media.xiph.org/video/derf/ and more will be added soon.

Varying viewing conditions is going to make comparison of results much
more difficult, and will require many more subjects to get statistically
significant results, if that is even possible. Even using the same
software on multiple platforms can produce very different results
(famously, even on the same platform, if you open a video in Windows
Media Player, pause it, and then open a second copy of the same video,
the second copy will often have brightness and contrast levels very
different from the first).

If you are comparing multiple codecs, you must also ensure that the
software is configured appropriately to ensure a fair comparison. For
example, post-processing is seldom enabled for Theora (the current
implementation is mostly unoptimized and fairly expensive at high
resolutions), but often enabled for MPEG4 codecs. Enabling
post-processing is not generally useful for evaluating the quality of a
format, as most codecs could equally well take advantage of the
post-processing methods of another codec. One possibility is to use only
pre-decoded, uncompressed videos produced with a carefully controlled
toolchain, however this will have difficulty fitting on a single DVD for
even a moderate number of sequences and codecs.