[vorbis-dev] Comparison between Ogg Vorbis and LAME

[Manuel Amador]

Guys (and girls),

I\'ve been testing Ogg Vorbis quality at home, and I thought I\'d share the 
results with you. For those who want the conclusion, Vorbis 1.0 RC3 sounds 
better than LAME 3.70 vbr/cbr (which I\'m told now that it\'s broken for VBR), at 
less bitrate, however, it\'s not perfect (and probably never will, since it\'s a 
lossy audio codec) and could use some help.  Further testing will yield better 
results.

Background:

Let me tell you that I\'ve been to discos and concerts and I think all that was 
bad to my ears since I can honestly tell that my hearing has diminished. I\'m 22 
however, and I can tell which one of the three TVs is on the minute I come 
home, even when they\'re muted.  I can also listen to 19 KHz sine waves, 
although at a rather loud volume that would burn the speakers on 100 Hz sine 
waves.  I have to ask people to repeat themselves in some cases, though. So I 
consider I have average ears, and in the spirit of open source and free 
software, I share these results with the hope they help.

Gear:

A self-made with off-the-shelf parts computer, with a Sound Blaster Live! OEM.  
I turned all audio input sources off since they introduce hiss, motorbike sound 
(CD input) and distortion, leaving only Wave selected.  I tried using a pair of 
$100 Technics headphones directly hooked to the headphone out of the sound 
card, but found that listening to the samples through an Aiwa NSX-V30 (el 
cheapo 30W sound system) was both easier and sounded moere natural, which made 
distinguishing sounds better.

After the test, I reconnected the rear speakers (which I didn\'t use), powered 
by a 1975 Sansui A-M7 amplifier.  I should have used the Sansui, if only 
because at its loudest volume you can hear NO hiss, while on the Aiwa you do 
(the Aiwa speakers are better than the Sansui\'s, with more frequency range, but 
they don\'t withstand as much power).  It\'s probably because of RF interference 
with the analog CD cable wired into the Aiwa\'s mainboard (it has a mainboard 
and the internal CD changer connects by a cable just like a CDROM audio 
cable!!!!).

I also have a Technics SU-Z2 1977 amp, which I\'d like to use but the amplifier 
is burnt, and when you raise the volume the left channel sounds terrible.  The 
speakers that came with that are fine, though, and sound amazingly good (and 
much louder, at 1.20 M tall with a 28cm woofer).  I might use the Sansui amp to 
power them.

Method:

For graphical analysis, I selected the entire waves in Cool Edit Pro 1.2 
(Alt+Z) and I scanned them with a window of 1024 and a Blackmann-Harris 
function.  I hope these are right.
For hearing, blind testing consisting of listening to the original sound, then 
listening to a randomized Winamp playlist of 10 samples, 5 originals and 5 
encoded, without seeing the list, and writing results down on a piece of paper.

I used a sample of the first 30 seconds of Aqua - Doctor Jones, directly ripped 
from the CD with EAC using C2 capabilities.  I encoded this sample using the 
default VBR mode in LAME 3.70 and quality levels 1, 3, 4, 5 and 10 in oggenc.

I tried to listen to the first part which consists of the sound of the forest, 
and at 5 seconds a string instrument attacks for the next 15 seconds.  It\'s 
there that I notice the differences.  Later on the song, when the dance part 
comes, I can\'t tell squat.  I\'ll try and describe the differences.

Results:

Let me tell you that it\'s possible to tell that the VBR MP3 is quite damaged in 
comparison to the original, missing high frequencies and having problems 
with \"underwater\" wavy sound.

The VBR MP3 done with -V 2 suffers from the same problems, except for the 
underwater sound.  Both MP3 sound like recorded on high-quality tape (a little 
wobbly due to the tape engine, and loss of high frequencies that is 
unavoidable).

Oggenc Q1 sounds distinctly like a 96 kbps Fraunhofer MP3, with a bit of the 
choppiness produced by stretching a sound on WaveLab without altering pitch.  
Oggenc beats MP3 on space.

Oggenc Q3 I could tell 10 out of 10 on the listening test.  It has a lack of 
definition both of the sound of the forest and the string instrument.  I 
noticed that on that part, bitrate is higher than on the dance parts (probably 
due to masking of frequencies on the loud dance part, that allows the encoder 
to \"throw away\" more music as it gets more complex and louder).

Oggenc Q4 I could tell 9 out of 10 times (three times in a row).  It has a very 
tiny lack of attack on the string instrument, but no difference on the ambience 
forest sound.  It\'s not perfect though.  It beats LAME hands down, at 133 kbps 
it sounds extremely better than an MP3.  I could archive at this quality and be 
happy, but I can tell the difference, and I\'m perfectionist.

Oggenc Q5 I told an average 7 out of 10 times (6/10,8/10,6/10) getting 
dangerously close to the \"toss a coin\" statistics (barring Murphy\'s laws of 
course).  Which makes Q5 a decent choice, except for the bitrate (170 kbps).  
Why does it need that much space on such a small quality increment?  If it 
weren\'t for the space, I\'d be encoding at this rate.  I expected like 145 kbps 
on this setting!

For Q10, I could have fared better at guessing on a baccarat table at the 
casino.  Good job!
What makes me wonder is why at q10 Oggenc does take some high frequencies away 
(graph analysis), which at 320 Kbps LAME does not, and the graph follows the 
original strictly.

Okay.  The graph analysis show that all Oggenc and LAME streams eat high 
frequencies for dinner, but Oggenc seems to do the task with surgical precision 
and a purpose.  See, I haven\'t told you yet that the song has a 17 KHz 
continuous sound (sort of like a string ensemble) at the first 20 seconds, so I 
suppose oggenc finds out that this sound masks all other high frequencies and 
throws them away.  I wonder if this takes \"fullness\" out of sound.  I didn\'t 
feel like it did.  

Conclusions:
* I\'m using Q5 for my encoding needs, but it takes too much space.  
Nevertheless, I had been using CBR 160 KBPS on LAME before that.
* Oggenc beats lame 3.70, both in quality and in bitrate.  I have to test 3.91.

<p>One thing: although MP3Pro is marketed as the next audio solution, I steer away 
from a \"We cut high frequencies then boost them on playback with a filter and a 
stream of hints\" solution.  I can tell too, and people are getting fooled. 
Correct me if I\'m wrong

One more thing: is there a lobby to exert pressure on hardware manufacturers to 
build Ogg Vorbis support into their hardware?  I\'d like to know what\'s taking 
them so long, and I\'m sure Thomson Multimedia had a lobby too.

Luck, great work and many thanks for this amazing software and the effort for 
ridding the world of software patents,

<p>   Manuel Amador (Rudd-O)
   http://www.usm.edu.ec/~amadorm/

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.