[vorbis-dev] end-user mode for a moment (side-by-side tests)

[Jelle Foks]

See my in-line comments.

Kenneth Arnold wrote:
> 
> Smack my curiosity, but I encoded some songs in Vorbis mode 2 and tried
> to be able to distinguish the difference between that and the uncompressed
> WAVs. (*smacks self*) but here's what I noted:
> 
> It's actually kind of hard to tell the difference :) (and I consider myself
> to have a decent set of ears, though not anywhere near the best)
> 
> I got my accuracy to about 90%, but I couldn't figure out what it was. Finally
> I figured out that it was some of the high-range in a few spots that hit with
> less -- brillience is the word that comes to mind -- than the uncompressed.
> Yeah this is 128k so what should I really expect, and compressed against
> uncompressed so almost no fault at all found here, but could the psycoacoustic
> model be tuned any? Maybe if somebody could assemble a "test kit" that a
> lot of people could use and try to tune the model to what they thought
> sounded best, then the results could be averaged? Or do we have it on higher
> authority that the psycoacoustics are the best they could be? (I am reminded
> of Linus Torvalds' announcement for 2.4.0-test2 on l-k back when I was
> subscribed.)
> 
> Wow. Not bad.
> 
> Now gotta try against MP3. Dang, does that mean I have to grab notlame or
> bladeenc? Darn... I didn't even install them when I reinstalled last,
> because it seems that I have my audio compression needs taken care of.
> 
> Back to developer mode:
> 
> Thanks, Ralph, for the Ogg todo sent a while ago. I've only now got to really
> studying it and looking at what to do. Looks like video is it. So a couple
> questions for the list:
> 
> 1. Where's the Tarkin source anyway?
> 
> 2. I am aware that Tarkin uses wavelets. MPEG uses object detection and
> motion estimation. What other methods are out there? Does anybody know
> of any new, cool methods for compressing video? Or, failing that, does
> anybody know [of] anyone who does?
> 
> 3. I have looked over the MPEG document Marshall said to look over a while
> ago (about varying levels of detail). I think that's a good idea (in fact
> that was a goal even before I read that). See what you all think about my
> personal codec wishlist (from a starting-from-scratch viewpoint, even though
> it probably won't work out that easily):
> 
> * Three levels: packet, frame, and field. Packet holds all the stuff that
>   should naturally go together and is otherwise worthless when split up.
>   (I'm thinking streaming here). Field is collection of packets that
>   describes part of a frame. It may pull information from a lot of sources,
>   e.g., raw image data, data from frames earlier / later (with an arbitrarily
>   adjustable window), "scratch" area, whatever. It should have the capability
>   to embody vector graphics, arbitrary transforms, effects, etc. even if the
>   encoder can't pick them out from a source video (if it could, that'd be
>   great, but that gets very compex). Maybe field == packet; I need to think
>   some more about that. But by "part of a frame", I mean a level of detail
>   as opposed to a region (although region might be useful also). Object
>   descriptions are hierarchical in importance by nature; the codec should
>   take advantage of this. Coding should be done residually, i.e., take as
>   much information about the frame as can be embodied relatively simply, and
>   repeat with what's left over. The amount of complexity per independent
>   block should be adjustable over a wide range. Each block iteration
>   (hierarchical level) could be assigned a priority, and when streaming, the
>   transport could choose to only send the blocks above priority x. Different
>   methods could be used to formulate these blocks, possibly even different
>   methods for different blocks describing the same area. This would allow
>   motion estimation to be used for entire objects, and e.g. wavelets for
>   details about the object. The definitions and implementations of the
>   residue and coding areas are left for later, to allow for more than
>   enough flexibility (I hope).
> * Every frame should be able to reference back to frames before it, i.e.,
>   no MPEG's I frames (except maybe at the beginning of the stream). 

If there are too many dependencies upon 'previous data', such as what
happens when you send/store I-type image data only very occasionally,
then you will have very slow or difficult seeking, channel zapping, etc.

>   Okay,
>   so maybe there should be I-frames, but use them more carefully. 

If 3d transforms are used, then there is not much need for something
like a I/P/B-type frame concept, because you're looking at muti-frame
data coefficients in the transformed domain. Here, the depth in time of
the 3d transform is similar to the 'I-frame frequency' in 2d-transform
coding.

the I-P-B frame types are a direct result of the current 2d transform
coding methods using predicive coding in the time domain. Back in the
old days, image compression method even did predictive coding in the
pixel domain, but when they stepped over towards transform coding, then
there was no need to keep doing that. The only place where prediction
remains is on the boundaries of the transforms: in 8x8 DCT coding (MPEG,
JPEG, H.26x), this is at the DC DCT coefficients plus in the time
domain.  In NxMxQ 3d-wavelet coding, prediction will only help at the
edges of the pixels and the group of frames that are transformed as a
whole.

>   Possibly
>   a lossless compression could be made from them... 

Lossless compression can be made from any compression method where the
residual entropy is sufficiently low. Lossless compression doesn't
require I-frames.

>   but back to the main
>   issue here: a typical viewer will be watching the video for at least 100
>   megabits before [s]he even starts to worry about quality as opposed to
>   content. 

Unless the viewer is receiving the stream over a 56k POTS modem or
similar. Even on 512kbit ADSL that is still more than 3 minutes.

>   So I-frames can be very sparse. 

I'de hate to be able to seek only to 3-minute or more intervals or wait
up to three minutes after each seek because the decoder needs to
reconstruct sufficient 'history'. Also I'd hate to be able to zap
through channels at only one channel per three minutes.

>   The tradeoff is more redundancy in the diff frames. 

Not completely. In your proposal all difference frames are changing the
reference frames, because each decompressed difference frame can be a
reference frame. In that case you have the problem of accumulated
errors. Especially in transform coding, where various implementations of
decoders may not be bit-true equal (due to various decoding
environments: processors, hardware (read: differences in rounding,
optimizations, efficiency and available data types)). After each
difference frame that is used as reference frame, the reference
available decoder deviates a bit more from the reference available in
the encoder, resulting in increased differences in the reconstructed
images.

>   Each diff frame should transmit the diff, plus some
>   data that the viewer should know if it's been watching since the last
>   I-frame. 
>   This would allow streaming to be able to take advantage of scene
>   similarity without worrying too much about the consequences of lost data.
>   Possibly the redundant data could have a temporal component attached also,
>   so when the video is saved to disk after streaming, it could be moved to
>   the proper place where it should have been first introduced and then
>   removed as much as possible to keep redundancy to a minimum on a fixed
>   medium (key point: the stream is not the compressed video. They work together
>   but both can be modified to hold the same or similar data in a more optimal
>   manner). Another key point: there's a lot you can tune here (amount of
>   redundant data transmitted, frequency of I-frames, etc.). More flexibilty.
> * VBR of course. But since streaming often works best when bitrate is constant
>   (TCP windows, if streaming over TCP), allow the redundant data to be filled
>   in whenever the data size is otherwise small.

If the bitstream can occasionally have a higher bit-rate than the
transmission medium, this results in latency (due to buffering).

Dropping frames is not a good solution here, because that is nothing
more than very bluntly reducing the VBR ceiling, which can better be
done inside the coding algorithm.

> * Scratch pad to save previous data. e.g. if scene is switching between two
>   talking heads, should save data associated with one when switching to other.

AFAIK, MPEG4 solves that by separating object descriptions and image
structure. In other words: in MPEG4, not all known and/or previously
known objects must be displayed at all times. This allows an encoder to
'keep' some objects across scene switches.

>   Key point is that maybe viewer didn't catch that old data; maybe send it
>   before stream starts playing, or put it in the redundant frames. First
>   sounds nice if you're not multicasting; second is more suited for
>   broadcasting.

It's all dependant on the application, many applications won't accept
the latency and other problems you get if you trade everything off
against maximum compression. The Ogg Video codec should be able to
produce the perfect stream for each application, but not every Ogg Video
stream has to be perfect for each application. Hence, keep all that
parametrizable, and keep as much of the details outside of the standard
and codec, let the application decide which parameters tickle it's
sweetspot. I think a video stream format is best kept simple: KISS (keep
it simple, stupid).

> * Assume viewer knows everything about the stream you sent, then either the
>   viewer could ask (unicast better again) or the streamer could just resend
>   anyway (multicast) the missing data.
> 
> Spewing a lot to myself above, and I really didn't mean to spew that much,
> but chew on it and tell me what you think. That's the product of probably
> about 15 minutes of mostly continuous thought that is very likely disjointed
> and missing some key information still locked somewhere in my head, so don't
> take it as written in anything but sand sprinkled in tide pools. It's also
> 11:00 PM local time, so I may have gone insane and not known about it.
> 
> The bit of judgement in me that hasn't gone to sleep yet is telling me that
> this is a good place to stop.
> 
> Kenneth
> 
> PS - I'm going to really like reading that when I'm more awake. It'll be fun.
> 
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> Ogg project homepage: http://www.xiph.org/ogg/
> To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
> containing only the word 'unsubscribe' in the body.  No subject is needed.
> Unsubscribe messages sent to the list will be ignored/filtered.

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.