[opus] Questions Regarding Opus Test Vectors

[Timothy B. Terriberry]

Wang, Chris wrote:
> Benjamin,
>
> Thanks for the prompt response.  Are there other recommended methods to
> verify encoder implementations?

There are few if any requirements placed on the encoder by the actual 
RFC, however there are a number of things you can do which provide 
various amounts of assurance your encoder is working as expected.

The first test I would run is to write out the final range coder state 
of each packet, as is done in the file format used by opus_demo, and 
then use opus_demo to decode the files. It will check to make sure the 
final range coder state of the decoder matches. With high probability 
this will tell you if the decoder is decoding the same bitstream symbols 
the encoder thought it encoded.

Slightly more complicated is to have the encoder resynthesize the coded 
audio from its internal state as it encodes, and compare that with what 
the decoder produces. In libopus this behavior can be enabled via the 
RESYNTH #define, at least for the CELT layer.

http://www.ietf.org/proceedings/80/slides/codec-4.pdf covers a number of 
additional software tests we run. Determining the quality of a 
correctly-functioning encoder requires subjective (listening tests) 
and/or objective metrics (PEAQ, PESQ, etc.).