[theora] A comparison of VP3, and two MPEG-4 variants

[Freun Laven]

> Not from a group of experts, but a group of layman yes. Experts can
>have
> preconceptions based on objective measures and can tie them to

Laymen can also have preconceptions.

People get used to what they've been watching, and they judge everything
by it.  And after a while, they think of those distortions etc. as being
normal.

For example, consider the difference between a flat screen monitor and a
highly curved CRT monitor.  When you switch to a flat screen after years
with a regular one, everything on the flat one looks distorted because
you are used to being displayed distorted and now there isn't any.  Some
people will even insist the monitor is bad and that it should be taken
back.  Until they get over it and become comfortable with the new, then
they look at the old and wonder how they ever tolerated it.

The same is true in audio.  I've got a couple of mp3 songs that weren't
encoded well, one of which was done at only 64kbps.  But I've gotten
used to them and that's now what I expect.  When I re-encoded them with
ogg, they didn't sound right, even though they are indeed much closer to
the original cd.  To me, that 64kbps mp3 sounds *better* than the ogg at
128kbps simply because I got used to it.

That's subjective testing.  And it's wrong because it's not actually
better.  I just got used to it.

<p>> MOS is the benchmark to which all objective measures are compared. To
any
> individual his subjective measure is the only one which counts ... how
then
> can you look at the big picture and declare subjective measures
meaningless?

Because it's *too* subjective.  There is so much variation that it can't
be trusted.

Most video compression methods are "good enough" in the sense they can
tolerably recreate the original video scene.  But that doesn't really
say anything about whether they are good, or whether they work well or
poor on different types of stuff, etc.  Or whether the user has simply
gotten used to the flaws.

<p>> > With video it's even worse.  And for somebody like myself, who has
> > eyesight problems, what I would consider to be 'good' would probably
>
> That is a rather extreme example, on average over all potential users
these

It's not extreme to *me* [grin].

But that is kind of the point... everybody sees things somewhat
differently.  And if you have two smallish groups of people doing
independant tests, you probably are not going to get results that are
very similar.  They are more likely to generate two fairly different
results.  (Just like in the audio compression tests.  Of all the ones
I've seen, the only thing that has been reasonably consistant is that
mp3 is at least *slightly* worse than the new methods.)

> kind of things even out. Although since with subjective tests you
usually
> have a rather small group your opinion would indeed probably not be
usefull
> to include :/

Well, I don't know.... for a good statistical sampling, you do
occasionally need a 'wild card' to shake things up. [grin]

<p>> > A purely subjective comparison is worthless.
>
> Actually it is the only comparison of value :) Indeed, the value of
> objective measures themselves is measured by how well they correlate
with
> subjective scores.

I can't entirely disagree with that... especially since lossy
compression is based on what can be removed without being subjectively
noticed...  Although some of that is based on objective analysis of how
human visual and/or audio systems work.

But pure subjective testing is not good enough, either.

<p><p><p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'theora-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.