[theora-dev] Support for layers and alpha channel?

[Dennis Pennekamp]

First of all congratulations on reaching alpha status.

I was wondering if you people are implementing multiple layer support 
into the codec. I don't know if it should be in the container (ogg) or 
codec (theora) but perhaps the syncing requires at least support in the 
codec.

For example a commercial: a cup of hot chocolate with steam escaping. 
Little action, low bandwidth. Then comes the rendered animated text 
across 'Hot chocolate'. Now the animation interferes with the static 
logo on the cup, complex patterns occur and bandwidth use shoots up.

Why? The text animation was an alpha channel put on top and might by 
itself even be lower bandwidth then the cup video. Perhaps if the two 
video's would remain alpha channel separated the total bandwidth would 
be lower.

Some more examples:

'Sorry, we're closed'
'Sorry, we zijn gesloten'
As a conversation goes on the director of a movie ups the suspense by 
letting the viewer read a card, a letter, or a report. While the 
subtitles are busy translating the conversation the on-screen text is 
lost in translation or reduced to a oneliner to fit in, even if done 
well it kind of breaks concentration as you must choose what to read. 
Would it not be cool to have the possibility to overlay the text of a 
scene with a localized version of the text. Using an alpha channel 
gives low bandwidth, expressive freedom and streaming choice. Yes, the 
stream could name the available language overlay channels and the 
streaming client could say no, I don't need that part of the stream. 
The overlay track could be called a translation overlay coded 'nl NL' 
and 'nl BE' meaning Dutch translation suited for the Netherlands and 
Belgium.

Fixing
Some people prefer to have the broadcast logo removed because of 
personal preference, because you have a licence to rebroadcast with 
your own logo, or because it interferes with subtitles. This may leave 
a smudge in the corner and some people may prefer to just see the logo 
instead, so the smudge could be on an alpha cannel and the viewer can 
decide to use it or not. A removed logo layer. It could also be a 
fixing layer, like the removal of the huge JVC logo in Highlander 
Endgame Special Edition. If good timecoding is available users could 
even share small overlay movies that fix visual bugs in already bought 
movies.

Underneath It All
Remember the No Doubt video where the singer straddles and orgasms and 
blinks of screen (to that happy place:-) to quickly return back on 
screen? Perhaps it would easier for the codec if she would be on a 
layer that simply jumps behind the background layer. Perhaps even more 
so if the motion in that layer can be frozen in time and continue when 
it is brought back to the front.

Also think about rougher videos where the drumsolo makes two different 
videos interweave like a flashlight, demanding complete screen redraws 
during the entire scene. Would it not be more efficient if you have two 
video streams where the player is instructed to franticly hustle them 
back to front with a simple rearrangement of the layer stack? Maybe, 
but definitely yes if the background video layer can be frozen in time.

If you allow continued play of background layers you could have 
integrated parallel video's. A guitar player sits by the camp fire 
singing his song about life, which comes to life in MTV style action 
sequences. Cool for an MTV broadcast but what if I want the video as 
live background picture on my plasma 'painting' on the wall when 
receiving visitors, I've only got so many MTV-live DVDs. Well if the 
layer of just the singer by the campfire would continue in the 
background and the composition would be named 'Default - Action' and 
'Romantic' I would have a great live painting on the wall by choosing 
'Romantic'.

We can all remember video's with scene reuse, well maybe not if you 
don't like electronic music. Would it not be efficient to let the 
composition track (sort of like a midi track) know that a scene will be 
reused and when a scene is the same as a previous scene. OK, you are 
making a streaming codec, but wait what if the scenes are so well 
documented in the stream that even though the stream itself is 
continuous and can be picked up at many intervals, the player can say 
'if you want, you can skip the upcoming scene (video layer) because I 
already have it cached and verified as complete' to streaming server. 
On the other hand the stored movie as a whole is smaller, while the 
streaming server knows to send duplicate scenes always by default as 
the client may have had trouble receiving the first time or maybe has 
just tuned in.

Censorship
Different cultures have different censorship preferences, American 
bible belt prefers not to see female breasts, Dutch start frowning at 
genitals and Japanese at pubic hair. Let's just say that MTV Europe is 
a strange mix of (un)censorship depending on from who they got a video 
and what the target market was. Khia loses lip sync because someone 
decided to put the uncensored lyrics under the USA videoclip. Why not 
have it all and let the player decide what to play. Multiple layers for 
censored and uncensored scenes, bandwidth saved by a composition track 
which tells what scenes / layers to show and what can be used in all 
movies displayed. Standardized naming of compositions so that a player 
can be set to not play obscenities or sound effects today.

Also alpha channel is cool by itself, imagine a loop of a butterfly 
flapping on the background of a website, no you grab the movie from 
Konqueror and drag it into a dir, while it keeps flapping! Can you 
resist taking it for a spin around the screen, of course not, it's 
cooler then quicktime movies that continue playing in the Mac OS X dock.

<p>So you are creating an open source alternative, great, my point is that 
an alternative becomes a lot more viable if it not only it has feature 
parity with better license structure but also has compelling new 
features. Which brings me to overscanning, wait this mail is just too 
big I'll send another one.

So what do you think? Am I a madman or do I have some good idea's. :-)
Idea's only BTW I can't code.

Greetings,
Dennis SCP

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'theora-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.