[vorbis-dev] Comparison between Ogg Vorbis and LAME

[Gian-Carlo Pascutto]

----- Original Message -----
From: "Manuel Amador" <amadorm at zeus.usm.edu.ec>
To: <vorbis-dev at xiph.org>
Sent: Wednesday, February 13, 2002 5:42 PM
Subject: [vorbis-dev] Comparison between Ogg Vorbis and LAME

<p>> Let me tell you that I\'ve been to discos and concerts and I think all
that was
> bad to my ears since I can honestly tell that my hearing has diminished.
I\'m 22
> however, and I can tell which one of the three TVs is on the minute I come
> home, even when they\'re muted.  I can also listen to 19 KHz sine waves,
> although at a rather loud volume that would burn the speakers on 100 Hz
sine
> waves.  I have to ask people to repeat themselves in some cases, though.
So I
> consider I have average ears, and in the spirit of open source and free
> software, I share these results with the hope they help.

A little note: your ability to hear a 'standalone' 19kHz wave has
little bearing on your actual ability to hear those frequencies
in music. If you want to get a more interesting test, pick a normal
song, and lowpass filter it (make sure to use a hq one to avoid aliasing
artifacts). I bet you'll have great trouble detecting even a 17kHz
lowpass in a blind test.

> For graphical analysis, I selected the entire waves in Cool Edit Pro 1.2
> (Alt+Z) and I scanned them with a window of 1024 and a Blackmann-Harris
> function.  I hope these are right.

They are wrong. Any _graphical_ analysis of a _perceptual_ codec
is wrong. It's what you hear that matters, not what you see. And not
what you _think_ you hear for that matter.

> For hearing, blind testing consisting of listening to the original sound,
then
> listening to a randomized Winamp playlist of 10 samples, 5 originals and 5
> encoded, without seeing the list, and writing results down on a piece of
paper.

This is acceptable, although 10 trials doesn't really provide much
statistical certainty in the general case.
Read up a little on ABX testing - you'll find it usefull for proper blind
testing.

> I used a sample of the first 30 seconds of Aqua - Doctor Jones, directly
ripped
> from the CD with EAC using C2 capabilities.  I encoded this sample using
the
> default VBR mode in LAME 3.70 and quality levels 1, 3, 4, 5 and 10 in
oggenc.
>
> I tried to listen to the first part which consists of the sound of the
forest,
> and at 5 seconds a string instrument attacks for the next 15 seconds.
It\'s
> there that I notice the differences.  Later on the song, when the dance
part
> comes, I can\'t tell squat.  I\'ll try and describe the differences.

Note that one sample is not saying much. You certainly shouldn't be making
any general-sounding conclusions from it.

> The VBR MP3 done with -V 2 suffers from the same problems, except for the
> underwater sound.  Both MP3 sound like recorded on high-quality tape (a
little
> wobbly due to the tape engine, and loss of high frequencies that is
> unavoidable).

MP3 VBR with LAME is tricky business. As you probably have noted by
now, LAME 3.70 is way outdated. Moreover, getting the best settings
is a tough job. To see the best MP3 is capable of right now, try
LAME 3.91 with the setting '--alt-preset standard' (and nothing
else) If you notice any imperfections with that setting, please mail
again. Many people will be quite interested :)

> Oggenc Q1 sounds distinctly like a 96 kbps Fraunhofer MP3, with a bit of
the
> choppiness produced by stretching a sound on WaveLab without altering
pitch.
> Oggenc beats MP3 on space.

Not sure what you mean here. Either you compare quality at equal bitrates,
or bitrates at equal quality. What are you doing here?

> Oggenc Q4 I could tell 9 out of 10 times (three times in a row).

In other words, you scored 27/30. There is a big statitical difference
between 9/10 and 27/30 :)

> Oggenc Q5 I told an average 7 out of 10 times (6/10,8/10,6/10) getting
> dangerously close to the \"toss a coin\" statistics (barring Murphy\'s
laws of
> course).

20/30 I'm too lazy to do the math, but this doesn't look like a sufficiently
significant result. (Someone correct me if...)

> I wonder if this takes \"fullness\" out of sound.  I didn\'t
> feel like it did.

If you couldn't tell the ogg from the original, it obviously didn't.

If you are interested in doing listening tests, I'd advise you to take a
look at http://ff123.net and http://hydrogenaudio.org
We regularly organize listening test, doing the best we can to make them
as accurate and objective as possible, and we can always need more trained
listeners.


--
GCP

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.