[theora-dev] Analogue artifact estimation

Mon Sep 2 15:35:00 PDT 2002

> What I'm thinking of is a true psycho-visual model that tracks the human
> eye/brain's ability to perceive things in various frequency/spatial
> domains, ie least noticeable differences and such.  I have yet to see any
> compelling research in this area that has been effectively reduced to
> practice in designing compression algorithms.

This would certainly be extremely cool, and likely have huge benefits, especially if a free software project were to get to this before anyone else. Of course, it would likely also involve a lot of work, but it sounds like we've got time and at least a few people who know what they're doing. 

This might be a good time time for me to go off-topic a little bit and ask about how this project is supposed to work. As I'm understanding it, and I may very well be totally wrong, the purpose of Theora is to create a way to put the VP3.2 video codec and the Vorbis audio codec into the Ogg container, thus creating an essentially patent-free audio and video format to compete with MPEG, Windows Media, RealMedia and the like. Meanwhile, there is another project, Tarkin (aka W3D) which is trying to create a totally new, multi-dimensional video codec, with a release date sometime in the far-off future. Tarkin, as it is, seems a bit dead in the water. 

What I guess I'm getting around to is this: Is this kind of work (psycho-visual models, filtering, etc.) on the video codec itself outside the scope of Theora? Is Theora really 2 projects, one for the container, one for the video codec? Do we want to make drastic changes to the codec, or should we hold off until Theora 1.0 is out the door? According to the FAQs:

"Theora will be almost entirely based upon the VP3 codec designed by On2." 

Do we really think that it will take a year to figure out how to put VP3 into the Ogg container? Isn't a lot of this work already done with the "OGM" format that is being used in the "DVD Back-Up" scene? (http://www.bunkus.org/videotools/ogmtools/)

Am I totally missing something here?

-Henry    

----- Original Message -----
From: "Daniel B. Miller" <dan at on2.com>
Date: Mon, 2 Sep 2002 16:28:12 -0400 (EDT)
To: <theora-dev at xiph.org>
Subject: Re: [theora-dev] Analogue artifact estimation

<p>> true enough -- the work in this area appears to have remained proprietary.
> My point was that in published academic research, which is where the
> newest ideas usually come from, pretty much the only objective measure
> ever cited is PSNR.  Companies such as Real and various MPEG encoders are
> probably doing more internally but are disinclined to share the research
> for business reasons, and possibly out of patent concerns.
> 
> However I still think that even what is being done behind closed doors is
> pretty rudimentary and ad-hoc.  I don't get the sense there is much of a
> rigorous intellectual foundation to any of this stuff.  Even vqeg type
> tests, which have been around a while (I think Sarnoff has products in
> this area too), tend to be very arbitrary -- a bunch of kernel comarators,
> edge detectors and so on.  Then they get people in a room and rate videos,
> comaring the subjective ratings to the 'objective' measures and trying to
> calibrate some sort of equivalence chart between them.  This is all fine
> from a practical point of view, but I still wish it was based on some
> first principles rather than being so empirical.
> 
> What I'm thinking of is a true psycho-visual model that tracks the human
> eye/brain's ability to perceive things in various frequency/spatial
> domains, ie least noticeable differences and such.  I have yet to see any
> compelling research in this area that has been effectively reduced to
> practice in designing compression algorithms.
> 
> 
>  ___  Dan Miller
> (++,) CTO and founder, On2 Technologies
> 
> On Sun, 1 Sep 2002, Henry Mason  wrote:
> 
> > There actually has been some work done on psycho-visual stuff. The XVID video codec has a "luminance-masking" function that applies more compression to parts of the picture that is darker, and RealVideo 9 seems to have some functions that allow it to compress the background of a picture more than the foreground. MPEG-4 also seems to have some specs for "GMC", which supposedly provides for some basic psycho-visual stuff.
> >
> > Unfortunately, in this business, research generally means patents. We may want to proceed with caution if we really want to get into this kind of thing.
> >
> > -Henry
> >
> > ----- Original Message -----
> > From: "Daniel B. Miller" <dan at on2.com>
> > Date: Sun, 1 Sep 2002 14:09:08 -0400 (EDT)
> > To: <theora-dev at xiph.org>
> > Subject: Re: [theora-dev] Analogue artifact estimation
> >
> >
> > > On Sun, 1 Sep 2002 jbradford at dial.pipex.com wrote:
> > >
> > > > > I think a lot these kind of artifacts are better handled through
> > > > > pre-processing.
> > > >
> > > > Yeah, that occurred to me, too, after I'd posted it :-).
> > > >
> > > > > Something like a temporal smoother can pretty effectively limit pixel
> > > > > crawl and the like, and it's a well known fact that simply blurring a
> > > > > video makes it compress more efficiently.
> > > >
> > > > Also, interestingly, adding a small amount of noise to a blurred video, (or still image), makes a lot of people percieve it as being sharper.  Of course, the noise-enhanced video is then practically useless for any further processing, but it's an interesting side-effect.
> > > >
> > > > > In fact, many have said that VP3 already pre- and post- filters too
> > > > > much, which can lead to blurry video.
> > > >
> > > > Perhaps an option to add noise after decompression might satisfy some
> > > > people!?  Sounds odd, but if it works...
> > > >
> > > hmm...  Ok, I'm the one who said we should focus on the bitstream, but
> > > since the subject of quality has come up I feel compelled to add my 2
> > > cents:
> > >
> > > I've recently done quite a bit of research into the whole issue of
> > > perceived vs. objectively measurable quality of compression algorithms.
> > > It turns out that the usual metric, PSNR (Peak Signal-to-Noise Ratio) is
> > > extremely limited for a number of reasons.  In particular, it gives little
> > > useful information about the frequency domain.  For example, an algorithm
> > > can improve PSNR by making the low-frequency information more accurate at
> > > the expense of high-frequency data.  This does little for perceived
> > > quality, and can actually make the result look worse.  Similarly, any
> > > noise at all reduces PSNR, whereas the human psycho-visual system is very
> > > good at extracting information from a noisy channel; this leads to codecs
> > > that avoid any noise at all costs, even when it would be an appropriate
> > > approximation of the original image (for instance, a textured wall will
> > > look better noisy than filtered).
> > >
> > > The point being, most algorithms including VP3 have been developed with
> > > PSNR being the major point of reference by which improvements are
> > > quantified and judged.  This has led to a generation of codecs that tend
> > > to blur out high frequency detail rather than allow any noise at all into
> > > the signal.
> > >
> > > I'm in the process of developing an alternative metric to PSNR that
> > > provides useful information about the frequency response of a given codec.
> > > It will also use an information-theoretical perspective to detect when
> > > signals are present within a noisy channel, rather than defining any noise
> > > at all as a negative.  These techniques could conceivably be used
> > > 'in-band' as well, ie we could rework the encoder to make various
> > > decisions (which motion vector to use, how to quantize DCT, etc) in a way
> > > that will better track perceived visual quality, rather than striving only
> > > to minimize the input/output error term regardless of frequency and noise
> > > characteristics.  Interestingly, most of this work tracks what is already
> > > done in audio; for some reason, the video crowd has shied away from any
> > > serious psycho-visual approach to compression.
> > >
> > > This approach could conceivably allow us to make substantial improvements
> > > in the VP3 encoder without having to make major changes in the bitstream
> > > definition.  We may even want to set up a separate discussion group,
> > > theora-encoder at xiph.org say, to focus on the issues of encoder quality as
> > > opposed to the more general task of defining the bitstream (in MPEG-land,
> > > this distinction is referred to as 'normative' vs. 'non-normative' work;
> > > normative meaning anything that affects the bitstream definition.)
> > >
> > > Please respond with your level of interest in this subject so we can
> > > calibrate the level of support we might have for such a project.
> > >
> > > -dbm
> > >
> > > > John.
> > >
> > >  ___  Dan Miller
> > > (++,) CTO and founder, On2 Technologies
> > >
> > > >
> > > > > ----- Original Message -----
> > > > > From: "Daniel B. Miller" <dan at on2.com>
> > > > > Date: Sat, 31 Aug 2002 18:31:43 -0400 (EDT)
> > > > > To: <theora-dev at xiph.org>
> > > > > Subject: Re: [theora-dev] Analogue artifact estimation
> > > > >
> > > > >
> > > > > > hi --
> > > > > >
> > > > > > Interesting points you bring up.  Realistically, theora is going to be
> > > > > > focused more on the definition of the bitstream and issues of
> > > > > > audio/video integration into the OGG format.  Advanced thoughts about
> > > > > > new codec ideas might be well received at the tarkin list
> > > > > > http://www.xiph.org/ogg/vmail.html.
> > > > > >
> > > > > > Not to discourage you from thinking about these things as they relate to
> > > > > > VP3;  however it's important to note that our primary objective
> > > > > > will be to define the bitstream before tackling major improvements in the
> > > > > > encoder.
> > > > > >
> > > > > >  ___  Dan Miller
> > > > > > (++,) CTO and founder, On2 Technologies
> > > > > >
> > > > > > On Wed, 28 Aug 2002 jbradford at dial.pipex.com wrote:
> > > > > >
> > > > > > > Hi List,
> > > > > > >
> > > > > > > Just batting a few ideas around here, but would it be possible to
> > > > > > > include in to the codec, estimation for common video artifacts that
> > > > > > > occur in the analogue world?
> > > > > > >
> > > > > > > For example, anything that's gone through a composite stage will
> > > > > > > likely have dot-crawl and false colour - if we can recognise this
> > > > > > > effect in the encoder, we can treat it as a special case.
> > > > > > >
> > > > > > > Other artifacts that come to mind are:
> > > > > > >
> > > > > > > * 3-2 Pull down
> > > > > > >
> > > > > > > * 30/25 and 25/30 FPS conversion artifacts
> > > > > > >
> > > > > > > * Dropouts, (both noisy, a-la VHS, and solid bars, a-la BetaSP, (which
> > > > > > > I believe halves the chroma resolution - can we possibly take this in
> > > > > > > to account as well?) )
> > > > > > >
> > > > > > > * Stobe effect of a CRT filmed by a different scan-rate camera
> > > > > > >
> > > > > > > * Colour changes in multi-layered painted cel animation, (where a
> > > > > > > character is, say, talking, and they repeatedly add and remove a cel
> > > > > > > from the top layer, their face flashes bright and dark :-) )
> > > > > > >
> > > > > > > Infact, existing video compression codecs do a *terrible* job of
> > > > > > > encoding hand painted animation, especially anime, despite claims by
> > > > > > > well known studios that it is not an issue.  Good motion estimation to
> > > > > > > compensate for poor registration of animation would be worth
> > > > > > > investigating.
> > > > > > >
> > > > > > > I think that the problem with all codecs to date is that they are
> > > > > > > designed to compress perfect material originating directly from a
> > > > > > > camera.  In my opinion, the above *analogue* artifacts are what give
> > > > > > > the digital codecs a hard time.
> > > > > > >
> > > > > > > I'd be very interested on any comments on these points.
> > > > > > >
> > > > > > > John. --- >8 ---- List archives:  http://www.xiph.org/archives/ Ogg
> > > > > > > project homepage: http://www.xiph.org/ogg/ To unsubscribe from this
> > > > > > > list, send a message to 'theora-dev-request at xiph.org' containing only
> > > > > > > the word 'unsubscribe' in the body.  No subject is needed. Unsubscribe
> > > > > > > messages sent to the list will be ignored/filtered.
> > > > > > >
> > > > > >
> > > > > > --- >8 ----
> > > > > > List archives:  http://www.xiph.org/archives/
> > > > > > Ogg project homepage: http://www.xiph.org/ogg/
> > > > > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > > > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > > > > Unsubscribe messages sent to the list will be ignored/filtered.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > _______________________________________________
> > > > > Get your free email from http://mymail.operamail.com
> > > > >
> > > > > Powered by Outblaze
> > > > > --- >8 ----
> > > > > List archives:  http://www.xiph.org/archives/
> > > > > Ogg project homepage: http://www.xiph.org/ogg/
> > > > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > > > Unsubscribe messages sent to the list will be ignored/filtered.
> > > > >
> > > >
> > > > --- >8 ----
> > > > List archives:  http://www.xiph.org/archives/
> > > > Ogg project homepage: http://www.xiph.org/ogg/
> > > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > > Unsubscribe messages sent to the list will be ignored/filtered.
> > > >
> > >
> > > --- >8 ----
> > > List archives:  http://www.xiph.org/archives/
> > > Ogg project homepage: http://www.xiph.org/ogg/
> > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > Unsubscribe messages sent to the list will be ignored/filtered.
> > >
> > >
> >
> >
> > --
> > _______________________________________________
> > Get your free email from http://mymail.operamail.com
> >
> > Powered by Outblaze
> > --- >8 ----
> > List archives:  http://www.xiph.org/archives/
> > Ogg project homepage: http://www.xiph.org/ogg/
> > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > Unsubscribe messages sent to the list will be ignored/filtered.
> >
> 
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> Ogg project homepage: http://www.xiph.org/ogg/
> To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> containing only the word 'unsubscribe' in the body.  No subject is needed.
> Unsubscribe messages sent to the list will be ignored/filtered.
> 
> 

-- 
_______________________________________________
Get your free email from http://mymail.operamail.com

Powered by Outblaze
--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.