[Vorbis-dev] Checking accuracy of the output

[nagaraja.sreerama at wipro.com]

Further to the tests, this time I used a sine sweep at 48000Hz and
encoded using the oggenc1.0.1, to get a ogg bitstream (attached). The
same ogg bitstream is passed to both oggdec(v1.0.1) and tremor code.
When compared the RMS value, the result is as follows:

File name 1: test2_sine_sweep_48k_tremor.pcm (16-bit output)---> tremor

File name 2: test2_sine_sweep_ref.pcm (16-bit output) ---> oggdec 1.0.1

Total number of samples  = 1440037
RMS error = 2.132247e-005
Allowed: Full Accuracy (16-bit SNR) = 8.81e-006
         Limited accuracy (12-bit SNR) = 1.409e-004
 SNR = 14.724785
Peak error = 3.051758e-005 (allowed[2^-14]=6.1035e-005)
Peak error = 256 levels (out of 24-bit)

Please note that according to the tests the 16-bits output should be
converted to 24-bit by forcing the last 8-bits to 0. The same is done
for both the reference output and the tremor output. Now its clear that
the tremor code is not fully compliant but with limited accuracy.

The same test is performed with two other .ogg songs. While one of them
gave almost similar results, the other song yielded drastic changes like
rms error of around 2.5e-1 etc.

Can anyone please comment on this ?

Best regards,
Nags..

-----Original Message-----
From: vorbis-dev-bounces at xiph.org [mailto:vorbis-dev-bounces at xiph.org]
On Behalf Of nagaraja.sreerama at wipro.com
Sent: Friday, January 28, 2005 5:09 PM

Hi,
But did anyone tried this test on the tremor code ?
I did a random check (of course not with any reference sine wave or so,
but with the input file as some audio song) and the result was shocking
when I tried the above test.
 Can anybody check up this point w.r.t. tremor code accuracy ?
Best regards,
Nags..

-----Original Message-----
From: vorbis-dev-bounces at xiph.org [mailto:vorbis-dev-bounces at xiph.org]
On Behalf Of John Ripley
Sent: Sunday, January 23, 2005 8:57 PM
To: Tal
Cc: vorbis-dev at xiph.org
Subject: Re: [Vorbis-dev] Checking accuracy of the output

I'd run a set of tests similar to the MP3 compliance tests used for MAD:
http://www.underbit.com/resources/mpeg/audio/compliance/
Take the difference signal of your output and the "ideal" output, then
calculate RMS and peak. A subtle thing here is that the ISO MP3 test
streams are not created by an encoder - the coefficients are directly
generated. You'll have to compare the accuracy of your decoder against
the accuracy of the reference Vorbis encoder + decoder combination.
Personally, I think the "fully compliant" criteria aren't nasty enough.
I would only accept an error due to truncation (+/-1 max). On the other
hand, I agree with the "limited accuracy" criteria: you can get away
with about 12 bit SNR without double blind tests picking up on the
difference. Just don't say that within earshot of a so called
"audiophile" :)

- John Ripley.



Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Mailadmin at wipro.com immediately
and destroy all copies of this message and any attachments.