[Vorbis-dev] Checking accuracy of the output

[nagaraja.sreerama at wipro.com]

Hi,
I agree with you that it would be better to run the compliance tests
similar to MP3. I also agree with you that given the lack of any
reference test vectors, we may have to depend on the Vorbis reference
Encoder and Vorbis reference Decoder (both floating point
implementations) as the reference for accuracy check of given decoder
using the above tests.
But did anyone tried this test on the tremor code ? Tremor code being a
Fixed point code, might have the loss of some bits compared to the
original implementation in floating point arithmetic. I did a random
check (of course not with any reference sine wave or so, but with the
input file as some audio song) and the result was shocking when I tried
the above test.
The procedure was like this:
I converted the song to a vorbis file using the reference vorbis
encoder. I passed the same file to (a) reference vorbis decoder and (b)
tremor code and got the outputs. Then tried to find the RMS difference
as per the procedure above (the same as the MP3 test suits). Surprising
to me, I got only a SNR of about 3 bits !! Probably something is wrong
in getting the results ?? Another observation is that through listening
test I could not make out the difference in quality (at least not much,
though I'm not a serious listner).
Can anybody check up this point w.r.t. tremor code accuracy ?
Best regards,
Nags..

-----Original Message-----
From: vorbis-dev-bounces at xiph.org [mailto:vorbis-dev-bounces at xiph.org]
On Behalf Of John Ripley
Sent: Sunday, January 23, 2005 8:57 PM
To: Tal
Cc: vorbis-dev at xiph.org
Subject: Re: [Vorbis-dev] Checking accuracy of the output

Michael Smith wrote:
 >>If you've made changes that produce (deliberately) different output,
>>then it's much, much more complex. You really need to do double-blind
>>listening tests to see whether your improvements are, in fact,
>>improvements.

Tal wrote:
> Hi,
> Thanks for the quick response,
> We did some optimizations that deliberately cut some corners.
> Is there a more accurate test to see if the the output is not hurt?

I'd run a set of tests similar to the MP3 compliance tests used for MAD:

http://www.underbit.com/resources/mpeg/audio/compliance/

Take the difference signal of your output and the "ideal" output, then
calculate RMS and peak. A subtle thing here is that the ISO MP3 test
streams are not created by an encoder - the coefficients are directly
generated. You'll have to compare the accuracy of your decoder against
the accuracy of the reference Vorbis encoder + decoder combination.

Personally, I think the "fully compliant" criteria aren't nasty enough.
I would only accept an error due to truncation (+/-1 max). On the other
hand, I agree with the "limited accuracy" criteria: you can get away
with about 12 bit SNR without double blind tests picking up on the
difference. Just don't say that within earshot of a so called
"audiophile" :)

(It's also a nice sanity check for work-in-progress code. At one point
while tweaking the MDCT, I'd accidentally reduced the quality of my
decoder to 8 bit SNR and removed everything from 11kHz to 22kHz. It took

me 2 weeks to notice...)

Are you reducing quality in order to save CPU/battery, or memory usage?

- John Ripley.
_______________________________________________
Vorbis-dev mailing list
Vorbis-dev at xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev



Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Mailadmin at wipro.com immediately
and destroy all copies of this message and any attachments.