[Vorbis-dev] Checking accuracy of the output

[John Ripley]

Michael Smith wrote:
 >>If you've made changes that produce (deliberately) different output,
 >>then it's much, much more complex. You really need to do double-blind
 >>listening tests to see whether your improvements are, in fact,
 >>improvements.

Tal wrote:
> Hi,
> Thanks for the quick response,
> We did some optimizations that deliberately cut some corners.  
> Is there a more accurate test to see if the the output is not hurt?

I'd run a set of tests similar to the MP3 compliance tests used for MAD:

http://www.underbit.com/resources/mpeg/audio/compliance/

Take the difference signal of your output and the "ideal" output, then 
calculate RMS and peak. A subtle thing here is that the ISO MP3 test 
streams are not created by an encoder - the coefficients are directly 
generated. You'll have to compare the accuracy of your decoder against 
the accuracy of the reference Vorbis encoder + decoder combination.

Personally, I think the "fully compliant" criteria aren't nasty enough. 
I would only accept an error due to truncation (+/-1 max). On the other 
hand, I agree with the "limited accuracy" criteria: you can get away 
with about 12 bit SNR without double blind tests picking up on the 
difference. Just don't say that within earshot of a so called 
"audiophile" :)

(It's also a nice sanity check for work-in-progress code. At one point 
while tweaking the MDCT, I'd accidentally reduced the quality of my 
decoder to 8 bit SNR and removed everything from 11kHz to 22kHz. It took 
me 2 weeks to notice...)

Are you reducing quality in order to save CPU/battery, or memory usage?

- John Ripley.