[theora-dev] Analogue artifact estimation

Mon Sep 2 13:28:12 PDT 2002

true enough -- the work in this area appears to have remained proprietary.
My point was that in published academic research, which is where the
newest ideas usually come from, pretty much the only objective measure
ever cited is PSNR.  Companies such as Real and various MPEG encoders are
probably doing more internally but are disinclined to share the research
for business reasons, and possibly out of patent concerns.

However I still think that even what is being done behind closed doors is
pretty rudimentary and ad-hoc.  I don't get the sense there is much of a
rigorous intellectual foundation to any of this stuff.  Even vqeg type
tests, which have been around a while (I think Sarnoff has products in
this area too), tend to be very arbitrary -- a bunch of kernel comarators,
edge detectors and so on.  Then they get people in a room and rate videos,
comaring the subjective ratings to the 'objective' measures and trying to
calibrate some sort of equivalence chart between them.  This is all fine
from a practical point of view, but I still wish it was based on some
first principles rather than being so empirical.

What I'm thinking of is a true psycho-visual model that tracks the human
eye/brain's ability to perceive things in various frequency/spatial
domains, ie least noticeable differences and such.  I have yet to see any
compelling research in this area that has been effectively reduced to
practice in designing compression algorithms.

<p> ___  Dan Miller
(++,) CTO and founder, On2 Technologies

On Sun, 1 Sep 2002, Henry Mason  wrote:

> There actually has been some work done on psycho-visual stuff. The XVID video codec has a "luminance-masking" function that applies more compression to parts of the picture that is darker, and RealVideo 9 seems to have some functions that allow it to compress the background of a picture more than the foreground. MPEG-4 also seems to have some specs for "GMC", which supposedly provides for some basic psycho-visual stuff.
>
> Unfortunately, in this business, research generally means patents. We may want to proceed with caution if we really want to get into this kind of thing.
>
> -Henry
>
> ----- Original Message -----
> From: "Daniel B. Miller" <dan at on2.com>
> Date: Sun, 1 Sep 2002 14:09:08 -0400 (EDT)
> To: <theora-dev at xiph.org>
> Subject: Re: [theora-dev] Analogue artifact estimation
>
>
> > On Sun, 1 Sep 2002 jbradford at dial.pipex.com wrote:
> >
> > > > I think a lot these kind of artifacts are better handled through
> > > > pre-processing.
> > >
> > > Yeah, that occurred to me, too, after I'd posted it :-).
> > >
> > > > Something like a temporal smoother can pretty effectively limit pixel
> > > > crawl and the like, and it's a well known fact that simply blurring a
> > > > video makes it compress more efficiently.
> > >
> > > Also, interestingly, adding a small amount of noise to a blurred video, (or still image), makes a lot of people percieve it as being sharper.  Of course, the noise-enhanced video is then practically useless for any further processing, but it's an interesting side-effect.
> > >
> > > > In fact, many have said that VP3 already pre- and post- filters too
> > > > much, which can lead to blurry video.
> > >
> > > Perhaps an option to add noise after decompression might satisfy some
> > > people!?  Sounds odd, but if it works...
> > >
> > hmm...  Ok, I'm the one who said we should focus on the bitstream, but
> > since the subject of quality has come up I feel compelled to add my 2
> > cents:
> >
> > I've recently done quite a bit of research into the whole issue of
> > perceived vs. objectively measurable quality of compression algorithms.
> > It turns out that the usual metric, PSNR (Peak Signal-to-Noise Ratio) is
> > extremely limited for a number of reasons.  In particular, it gives little
> > useful information about the frequency domain.  For example, an algorithm
> > can improve PSNR by making the low-frequency information more accurate at
> > the expense of high-frequency data.  This does little for perceived
> > quality, and can actually make the result look worse.  Similarly, any
> > noise at all reduces PSNR, whereas the human psycho-visual system is very
> > good at extracting information from a noisy channel; this leads to codecs
> > that avoid any noise at all costs, even when it would be an appropriate
> > approximation of the original image (for instance, a textured wall will
> > look better noisy than filtered).
> >
> > The point being, most algorithms including VP3 have been developed with
> > PSNR being the major point of reference by which improvements are
> > quantified and judged.  This has led to a generation of codecs that tend
> > to blur out high frequency detail rather than allow any noise at all into
> > the signal.
> >
> > I'm in the process of developing an alternative metric to PSNR that
> > provides useful information about the frequency response of a given codec.
> > It will also use an information-theoretical perspective to detect when
> > signals are present within a noisy channel, rather than defining any noise
> > at all as a negative.  These techniques could conceivably be used
> > 'in-band' as well, ie we could rework the encoder to make various
> > decisions (which motion vector to use, how to quantize DCT, etc) in a way
> > that will better track perceived visual quality, rather than striving only
> > to minimize the input/output error term regardless of frequency and noise
> > characteristics.  Interestingly, most of this work tracks what is already
> > done in audio; for some reason, the video crowd has shied away from any
> > serious psycho-visual approach to compression.
> >
> > This approach could conceivably allow us to make substantial improvements
> > in the VP3 encoder without having to make major changes in the bitstream
> > definition.  We may even want to set up a separate discussion group,
> > theora-encoder at xiph.org say, to focus on the issues of encoder quality as
> > opposed to the more general task of defining the bitstream (in MPEG-land,
> > this distinction is referred to as 'normative' vs. 'non-normative' work;
> > normative meaning anything that affects the bitstream definition.)
> >
> > Please respond with your level of interest in this subject so we can
> > calibrate the level of support we might have for such a project.
> >
> > -dbm
> >
> > > John.
> >
> >  ___  Dan Miller
> > (++,) CTO and founder, On2 Technologies
> >
> > >
> > > > ----- Original Message -----
> > > > From: "Daniel B. Miller" <dan at on2.com>
> > > > Date: Sat, 31 Aug 2002 18:31:43 -0400 (EDT)
> > > > To: <theora-dev at xiph.org>
> > > > Subject: Re: [theora-dev] Analogue artifact estimation
> > > >
> > > >
> > > > > hi --
> > > > >
> > > > > Interesting points you bring up.  Realistically, theora is going to be
> > > > > focused more on the definition of the bitstream and issues of
> > > > > audio/video integration into the OGG format.  Advanced thoughts about
> > > > > new codec ideas might be well received at the tarkin list
> > > > > http://www.xiph.org/ogg/vmail.html.
> > > > >
> > > > > Not to discourage you from thinking about these things as they relate to
> > > > > VP3;  however it's important to note that our primary objective
> > > > > will be to define the bitstream before tackling major improvements in the
> > > > > encoder.
> > > > >
> > > > >  ___  Dan Miller
> > > > > (++,) CTO and founder, On2 Technologies
> > > > >
> > > > > On Wed, 28 Aug 2002 jbradford at dial.pipex.com wrote:
> > > > >
> > > > > > Hi List,
> > > > > >
> > > > > > Just batting a few ideas around here, but would it be possible to
> > > > > > include in to the codec, estimation for common video artifacts that
> > > > > > occur in the analogue world?
> > > > > >
> > > > > > For example, anything that's gone through a composite stage will
> > > > > > likely have dot-crawl and false colour - if we can recognise this
> > > > > > effect in the encoder, we can treat it as a special case.
> > > > > >
> > > > > > Other artifacts that come to mind are:
> > > > > >
> > > > > > * 3-2 Pull down
> > > > > >
> > > > > > * 30/25 and 25/30 FPS conversion artifacts
> > > > > >
> > > > > > * Dropouts, (both noisy, a-la VHS, and solid bars, a-la BetaSP, (which
> > > > > > I believe halves the chroma resolution - can we possibly take this in
> > > > > > to account as well?) )
> > > > > >
> > > > > > * Stobe effect of a CRT filmed by a different scan-rate camera
> > > > > >
> > > > > > * Colour changes in multi-layered painted cel animation, (where a
> > > > > > character is, say, talking, and they repeatedly add and remove a cel
> > > > > > from the top layer, their face flashes bright and dark :-) )
> > > > > >
> > > > > > Infact, existing video compression codecs do a *terrible* job of
> > > > > > encoding hand painted animation, especially anime, despite claims by
> > > > > > well known studios that it is not an issue.  Good motion estimation to
> > > > > > compensate for poor registration of animation would be worth
> > > > > > investigating.
> > > > > >
> > > > > > I think that the problem with all codecs to date is that they are
> > > > > > designed to compress perfect material originating directly from a
> > > > > > camera.  In my opinion, the above *analogue* artifacts are what give
> > > > > > the digital codecs a hard time.
> > > > > >
> > > > > > I'd be very interested on any comments on these points.
> > > > > >
> > > > > > John. --- >8 ---- List archives:  http://www.xiph.org/archives/ Ogg
> > > > > > project homepage: http://www.xiph.org/ogg/ To unsubscribe from this
> > > > > > list, send a message to 'theora-dev-request at xiph.org' containing only
> > > > > > the word 'unsubscribe' in the body.  No subject is needed. Unsubscribe
> > > > > > messages sent to the list will be ignored/filtered.
> > > > > >
> > > > >
> > > > > --- >8 ----
> > > > > List archives:  http://www.xiph.org/archives/
> > > > > Ogg project homepage: http://www.xiph.org/ogg/
> > > > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > > > Unsubscribe messages sent to the list will be ignored/filtered.
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > _______________________________________________
> > > > Get your free email from http://mymail.operamail.com
> > > >
> > > > Powered by Outblaze
> > > > --- >8 ----
> > > > List archives:  http://www.xiph.org/archives/
> > > > Ogg project homepage: http://www.xiph.org/ogg/
> > > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > > Unsubscribe messages sent to the list will be ignored/filtered.
> > > >
> > >
> > > --- >8 ----
> > > List archives:  http://www.xiph.org/archives/
> > > Ogg project homepage: http://www.xiph.org/ogg/
> > > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > > Unsubscribe messages sent to the list will be ignored/filtered.
> > >
> >
> > --- >8 ----
> > List archives:  http://www.xiph.org/archives/
> > Ogg project homepage: http://www.xiph.org/ogg/
> > To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> > containing only the word 'unsubscribe' in the body.  No subject is needed.
> > Unsubscribe messages sent to the list will be ignored/filtered.
> >
> >
>
>
> --
> _______________________________________________
> Get your free email from http://mymail.operamail.com
>
> Powered by Outblaze
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> Ogg project homepage: http://www.xiph.org/ogg/
> To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
> containing only the word 'unsubscribe' in the body.  No subject is needed.
> Unsubscribe messages sent to the list will be ignored/filtered.
>

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.