[vorbis-dev] video compression using textured polygons?

[Kenneth C. Arnold]

On Sat, Jan 06, 2001 at 01:43:20PM +0330, jsr at dds.nl wrote:
> 
> I was just thinking last night, how about this:

Weird, so was I :) and I'm not just saying that to sound
smart... actually I have evidence saying that I thought about this
Thursday at 10:05 AM EST, but that's very much besides the point.

> For each frame, detect edges and divide the frame into a 
> bunch of non-overlapping polygons. Extract textures from 
> the first frame, then save changes in polygon vertex 
> coordinates only, unless the textures change too much in 
> which case new textures (which could be wavelet compressed) 
> would be stored.
> 
> Also, if the movements of the vertices would be taken 
> across multiple frames they could be expressed as splines, 
> causing more compression.

The problem here is of course the encoder. Do you know any algorithms
that could possibly come close to doing this in anything near
real-time? That's what discourages me from putting it in the core of a
video spec (though you could have one polygon filling the entire
screen and the texture changing, but that would not compress well in
the proposed scheme).

The textures themselves could change, but in relatively predictable
ways. For example, consider (as Sean Lynch was talking to me about) a
road with cars passing by. Every time a car moves, a conventional
video codec would have to retransmit the road underneath. A
polygon- (or generic object) based codec could realize that the road
is now mostly the same as it was before the car passed, and just tell
the player that object "car" moved, and (if it doesn't already know)
polygon "road"'s texture has changed to <what the road was underneath>

> I realise that this is quite a lot like mpeg 4, with its 
> sprites and body objects, but I think it's more generic. 
> And it has the added bonus of being able to do playback 
> hardware-accelerated on standard 3D hardware.

I don't know how MPEG4 works here, but I will say that a simple
polygon would not cut it in all cases. The encoder will probably not
be able to handle it, but the codec should provide for entirely
arbitrary 3d worlds (awesome for recording a quake level to show your
friends :), though in a way that doesn't overly harm the compressino
for the rest of the encoder. Same for any other encoder extensions. I
will be playing with this sort of stuff soon in a prototype
renderer-based player (encode by hand for a while) and keep you posted
on how things work out.

> Problems I foresee: edges may look too sharp, or if the 
> polygons don't have enough edges too straight. Also the 
> frame-to-frame matching of polygons absolutely needs to be 
> correct or things go wrong spectacularly.

Encoder issues... the player for this would be absurdly simple on a
system with 3d accel (or a port of Mesa).

> Well, any thoughts are welcome.

Chirplets? Can anyone go beyond the buzzword and tell me what's going
on with them? Everything I've seen looks promising but isn't enough to
write an actual implementation or even do something by hand with them.

> Lourens

Always happy to talk to smart people.


-- 
Kenneth Arnold <ken at arnoldnet.net> / kcarnold / Linux user #180115
http://arnoldnet.net/~kcarnold/



<HR NOSHADE>
<UL>
<LI>application/pgp-signature attachment: stored
</UL>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/octet-stream
Size: 233 bytes
Desc: not available
Url : http://lists.xiph.org/pipermail/vorbis-dev/attachments/20010107/62382e4c/part.obj