[Theora-dev] A/V sync strategies

[Keenan Pepper]

I tried to encode some video from a DVD with encoder_example and 
ffmpeg2theora, and it looked great, except that with both encoders the 
A/V sync would drift noticeably after about 10 minutes, and by the end 
of a two hour movie it would be a couple seconds off. So instead of just 
whining about this and waiting for someone else to fix it, I decided to 
try to learn about what's causing the problem and maybe even fix it 
myself. Being a newbie, I'll just explain my understanding of it and you 
experts can point out any misunderstandings and enlighten me.

Theora and MPEG2 handle A/V sync in different ways. Theora has an 
absolutely fixed frame rate and the time at which a frame should be 
displayed can be determined by multiplying the frame rate by the number 
of frames in the stream so far. That's why the header has an exact 
rational frame rate. MPEG video's frame rate can vary slightly because 
each frame has an associated PTS ("presentation timestamp"), which do 
not form an exact arithmetical sequence.

Is this right so far?

So it seems to me the only way to keep A/V sync when transcoding from 
MPEG or any timestamp-based format into Theora is to drop frames or 
alter audio samples when the A/V offset gets too high. Neither the 
encoder_example nor ffmpeg2theora currently does this, so the sync drifts.

If this is a correct description of the situation, I'll start hacking on 
ffmpeg2theora to implement some kind of sync-adjusting method. 
(Obviously it would be impossible with encoder_example because that only 
accepts raw audio and video with no sync information such as PTSs.)

Keenan Pepper