[Vorbis-dev] Checking accuracy of the output

Tue Feb 1 03:53:40 PST 2005

Further to the tests, this time I used a sine sweep at 48000Hz and
encoded using the oggenc1.0.1, to get a ogg bitstream (attached).
The same ogg bitstream is passed to both oggdec(v1.0.1) and tremor code.
When compared the RMS value, the result is as follows:

File name 1: test2_sine_sweep_48k_tremor.pcm (16-bit output)---> tremor

File name 2: test2_sine_sweep_ref.pcm (16-bit output) ---> oggdec 1.0.1

Total number of samples  = 1440037
RMS error = 2.132247e-005
Allowed: Full Accuracy (16-bit SNR) = 8.81e-006
         Limited accuracy (12-bit SNR) = 1.409e-004
 SNR = 14.724785
Peak error = 3.051758e-005 (allowed[2^-14]=6.1035e-005)
Peak error = 256 levels (out of 24-bit)

Please note that according to the tests the 16-bits output should be
converted to 24-bit by forcing the last 8-bits to 0. The same is done
for both the reference output and the tremor output. Now its clear that
the tremor code is not fully compliant but with limited accuracy.

The same test is performed with two other .ogg songs. While one of them
gave almost similar results, the other song yielded drastic changes like
rms error of around 2.5e-1 etc.

Can anyone please comment on this ?

Best regards,
Nags..

-----Original Message-----
From: vorbis-dev-bounces at xiph.org [mailto:vorbis-dev-bounces at xiph.org]
On Behalf Of nagaraja.sreerama at wipro.com
Sent: Friday, January 28, 2005 5:09 PM
To: jripley at rioaudio.com; shabi at t2.technion.ac.il
Cc: vorbis-dev at xiph.org
Subject: RE: [Vorbis-dev] Checking accuracy of the output

Hi,
I agree with you that it would be better to run the compliance tests
similar to MP3. I also agree with you that given the lack of any
reference test vectors, we may have to depend on the Vorbis reference
Encoder and Vorbis reference Decoder (both floating point
implementations) as the reference for accuracy check of given decoder
using the above tests. But did anyone tried this test on the tremor code
?
Tremor code being a Fixed point code, might have the loss of some bits
compared to the original implementation in floating point arithmetic. I
did a random check (of course not with any reference sine wave or so,
but with the input file as some audio song) and the result was shocking
when I tried the above test.
The procedure was like this:
I converted the song to a vorbis file using the reference vorbis
encoder. I passed the same file to (a) reference vorbis decoder and (b)
tremor code and got the outputs. Then tried to find the RMS difference
as per the procedure above (the same as the MP3 test suits).
Surprising to me, I got only a SNR of about 3 bits !! Probably something
is wrong in getting the results ?? Another observation is that through
listening test I could not make out the difference in quality (at least
not much, though I'm not a serious listner). Can anybody check up this
point w.r.t. tremor code accuracy ?
Best regards,
Nags..

-----Original Message-----
From: vorbis-dev-bounces at xiph.org [mailto:vorbis-dev-bounces at xiph.org]
On Behalf Of John Ripley
Sent: Sunday, January 23, 2005 8:57 PM
To: Tal
Cc: vorbis-dev at xiph.org
Subject: Re: [Vorbis-dev] Checking accuracy of the output

Michael Smith wrote:
 >>If you've made changes that produce (deliberately) different output,
>>then it's much, much more complex. You really need to do double-blind
>>listening tests to see whether your improvements are, in fact,
>>improvements.

Tal wrote:
> Hi,
> Thanks for the quick response,
> We did some optimizations that deliberately cut some corners. Is there

> a more accurate test to see if the the output is not hurt?

I'd run a set of tests similar to the MP3 compliance tests used for MAD:

http://www.underbit.com/resources/mpeg/audio/compliance/

Take the difference signal of your output and the "ideal" output, then

calculate RMS and peak. A subtle thing here is that the ISO MP3 test

streams are not created by an encoder - the coefficients are directly

generated. You'll have to compare the accuracy of your decoder against

the accuracy of the reference Vorbis encoder + decoder combination.

Personally, I think the "fully compliant" criteria aren't nasty enough.

I would only accept an error due to truncation (+/-1 max). On the other

hand, I agree with the "limited accuracy" criteria: you can get away

with about 12 bit SNR without double blind tests picking up on the

difference. Just don't say that within earshot of a so called

"audiophile" :)

(It's also a nice sanity check for work-in-progress code. At one point

while tweaking the MDCT, I'd accidentally reduced the quality of my

decoder to 8 bit SNR and removed everything from 11kHz to 22kHz. It took

me 2 weeks to notice...)

Are you reducing quality in order to save CPU/battery, or memory usage?

- John Ripley.
_______________________________________________
Vorbis-dev mailing list
Vorbis-dev at xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev

Confidentiality Notice

The information contained in this electronic message and any attachments
to this message are intended for the exclusive use of the addressee(s)
and may contain confidential or privileged information. If you are not
the intended recipient, please notify the sender at Wipro or
Mailadmin at wipro.com immediately and destroy all copies of this message
and any attachments. _______________________________________________
Vorbis-dev mailing list
Vorbis-dev at xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev

Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Mailadmin at wipro.com immediately
and destroy all copies of this message and any attachments.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test2_sine_sweep_48k.ogg
Type: application/octet-stream
Size: 66780 bytes
Desc: test2_sine_sweep_48k.ogg
Url : http://lists.xiph.org/pipermail/vorbis-dev/attachments/20050201/d8283a61/test2_sine_sweep_48k-0001.obj