[theora] A comparison of VP3, and two MPEG-4 variants

[Marco Al]

From: "Freun Laven" <FreunLaven at earthlink.net>

> Laymen can also have preconceptions.
>
> People get used to what they've been watching, and they judge everything
> by it.  And after a while, they think of those distortions etc. as being
> normal.

They will compare it against what we are trying to get close to, broadcast
quality video (and ultimately reality, people are perfectly able to
appreciate the quality of high-definition/high-refresh video for instance).
So not a big problem.

> The same is true in audio.  I've got a couple of mp3 songs that weren't
> encoded well, one of which was done at only 64kbps.  But I've gotten
> used to them and that's now what I expect.  When I re-encoded them with
> ogg, they didn't sound right, even though they are indeed much closer to
> the original cd.  To me, that 64kbps mp3 sounds *better* than the ogg at
> 128kbps simply because I got used to it.

Then for you it is better. Although given that you placed yourself in a
rather small minority again you disqualified yourself again for quality
tests, at least using that particular song.

> That's subjective testing.  And it's wrong because it's not actually
> better.  I just got used to it.

Even though I think all your objections are far fetched ... objective
measures dont solve this. You need to take an extra step for PSNR to be
meaningfull by itself, you have to believe that that after long time viewing
under given conditions PSNR starts correlating better with MOS after
adjustment than the MOS taken before does with it.

It is rather much to take on faith. Also given existing testing procedures
for video quality you once again you find yourself in a rather small
minority :)

> Because it's *too* subjective.  There is so much variation that it can't
> be trusted.

The variation of quality/PSNR for individuals is greater still. The chance
that a MOS score will predict my assesment of quality better than PSNR is
very high, to me that makes the MOS score more valuable than PSNR.

> But that is kind of the point... everybody sees things somewhat
> differently.

Our similarities far outnumber our differences.

> And if you have two smallish groups of people doing
> independant tests, you probably are not going to get results that are
> very similar.

They are going to be more similar to eachother than either is similar to
PSNR. If you think their tests are worthless, then what does that make the
PSNR?

> > Actually it is the only comparison of value :) Indeed, the value of
> > objective measures themselves is measured by how well they correlate
> with
> > subjective scores.
>
> I can't entirely disagree with that...

Would be hard to, given how easily it is to look up the VQEG website :)

> especially since lossy
> compression is based on what can be removed without being subjectively
> noticed...  Although some of that is based on objective analysis of how
> human visual and/or audio systems work.

Most researchers take physiology into account in the hope it will make their
measures better with subjective measures (although not all, check out some
of Zhou Wang's work for instance). When it turns out it doesnt their measure
will still get discarded regardless, if not by them ... then by everyone
else, look at VQEG testing for instance, because ultimately we have no
choice but to trust subjective measures as the ultimate bencharm. All we can
do is make the testing methodologies as realistic as possible.

> But pure subjective testing is not good enough, either.

If it is not good enough, then nothing is good.

Marco

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'theora-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.