[vorbis-dev] Comparison between Ogg Vorbis and LAME

[Manuel Amador]

I just followed whatever procedures I could think of to make sound comparison 
impartial.  It was a homemade test, yes.  But it still shows that LAME is 
easily beaten by ogg (even 3.91.  Alt presets produce way too much bitrate).

luck,

Quoting Gian-Carlo Pascutto <gcp at sjeng.org>:

> ----- Original Message -----
> From: \"Manuel Amador\" <amadorm at zeus.usm.edu.ec>
> To: <vorbis-dev at xiph.org>
> Sent: Wednesday, February 13, 2002 5:42 PM
> Subject: [vorbis-dev] Comparison between Ogg Vorbis and LAME
> 
> 
> > Let me tell you that I\\\'ve been to discos and concerts and I think
> all
> that was
> > bad to my ears since I can honestly tell that my hearing has
> diminished.
> I\\\'m 22
> > however, and I can tell which one of the three TVs is on the minute I
> come
> > home, even when they\\\'re muted.  I can also listen to 19 KHz sine
> waves,
> > although at a rather loud volume that would burn the speakers on 100
> Hz
> sine
> > waves.  I have to ask people to repeat themselves in some cases,
> though.
> So I
> > consider I have average ears, and in the spirit of open source and
> free
> > software, I share these results with the hope they help.
> 
> A little note: your ability to hear a \'standalone\' 19kHz wave has
> little bearing on your actual ability to hear those frequencies
> in music. If you want to get a more interesting test, pick a normal
> song, and lowpass filter it (make sure to use a hq one to avoid
> aliasing
> artifacts). I bet you\'ll have great trouble detecting even a 17kHz
> lowpass in a blind test.
> 
> > For graphical analysis, I selected the entire waves in Cool Edit Pro
> 1.2
> > (Alt+Z) and I scanned them with a window of 1024 and a
> Blackmann-Harris
> > function.  I hope these are right.
> 
> They are wrong. Any _graphical_ analysis of a _perceptual_ codec
> is wrong. It\'s what you hear that matters, not what you see. And not
> what you _think_ you hear for that matter.
I know that.  But for archival quality, it does.

> 
> > For hearing, blind testing consisting of listening to the original
> sound,
> then
> > listening to a randomized Winamp playlist of 10 samples, 5 originals
> and 5
> > encoded, without seeing the list, and writing results down on a piece
> of
> paper.
> 
> This is acceptable, although 10 trials doesn\'t really provide much
> statistical certainty in the general case.
> Read up a little on ABX testing - you\'ll find it usefull for proper
> blind
> testing.
> 
> > I used a sample of the first 30 seconds of Aqua - Doctor Jones,
> directly
> ripped
> > from the CD with EAC using C2 capabilities.  I encoded this sample
> using
> the
> > default VBR mode in LAME 3.70 and quality levels 1, 3, 4, 5 and 10
> in
> oggenc.
> >
> > I tried to listen to the first part which consists of the sound of
> the
> forest,
> > and at 5 seconds a string instrument attacks for the next 15
> seconds.
> It\\\'s
> > there that I notice the differences.  Later on the song, when the
> dance
> part
> > comes, I can\\\'t tell squat.  I\\\'ll try and describe the differences.
> 
> Note that one sample is not saying much. You certainly shouldn\'t be
> making
> any general-sounding conclusions from it.
I know that.  In this case, it sounds that way.
> 
> > The VBR MP3 done with -V 2 suffers from the same problems, except for
> the
> > underwater sound.  Both MP3 sound like recorded on high-quality tape
> (a
> little
> > wobbly due to the tape engine, and loss of high frequencies that is
> > unavoidable).
> 
> MP3 VBR with LAME is tricky business. As you probably have noted by
> now, LAME 3.70 is way outdated. Moreover, getting the best settings
> is a tough job. To see the best MP3 is capable of right now, try
> LAME 3.91 with the setting \'--alt-preset standard\' (and nothing
> else) If you notice any imperfections with that setting, please mail
> again. Many people will be quite interested :)
Alt preset sucks.  It produces way too high bitrate for me to be useful.  True, 
it is indistinguishable from the original in my case (not thruly tested) but 
nevertheless too much bitrate.  I\'d rather stay with oggenc -q 4.99.  Hey.  I 
tested q4.99.  I found that it\'s still not perfect, perhaps due to the guitar 
sounds being on the left channel only.

<p>> 
> > Oggenc Q1 sounds distinctly like a 96 kbps Fraunhofer MP3, with a bit
> of
> the
> > choppiness produced by stretching a sound on WaveLab without
> altering
> pitch.
> > Oggenc beats MP3 on space.
> 
> Not sure what you mean here. Either you compare quality at equal
> bitrates,
> or bitrates at equal quality. What are you doing here?
I\'m comparing perceptual quality.  The sound of oggenc q1 reminds me of a 96 
kbps mp3.

> 
> > Oggenc Q4 I could tell 9 out of 10 times (three times in a row).
> 
> In other words, you scored 27/30. There is a big statitical difference
> between 9/10 and 27/30 :)
> 
> > Oggenc Q5 I told an average 7 out of 10 times (6/10,8/10,6/10)
> getting
> > dangerously close to the \\\"toss a coin\\\" statistics (barring
> Murphy\\\'s
> laws of
> > course).
> 
> 20/30 I\'m too lazy to do the math, but this doesn\'t look like a
> sufficiently
> significant result. (Someone correct me if...)
> 
> > I wonder if this takes \\\"fullness\\\" out of sound.  I didn\\\'t
> > feel like it did.
> 
> If you couldn\'t tell the ogg from the original, it obviously didn\'t.
> 
> If you are interested in doing listening tests, I\'d advise you to take
> a
> look at http://ff123.net and http://hydrogenaudio.org
> We regularly organize listening test, doing the best we can to make
> them
> as accurate and objective as possible, and we can always need more
> trained
> listeners.
> 
> --
> GCP
> 
> 
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> Ogg project homepage: http://www.xiph.org/ogg/
> To unsubscribe from this list, send a message to
> \'vorbis-dev-request at xiph.org\'
> containing only the word \'unsubscribe\' in the body.  No subject is
> needed.
> Unsubscribe messages sent to the list will be ignored/filtered.
> 

<p><p>   Manuel Amador (Rudd-O)
   http://www.usm.edu.ec/~amadorm/IUI\'ll folow up later.  gotta go to class.

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.