[foms] Proposal: adaptive streaming using open codecs

[Jeroen Wijering]

On Oct 28, 2010, at 7:00 PM, Mark Watson wrote:

>>> It would be really great if the whole thing could run independently for audio and video. They can be completely decoupled for streaming and synchronized at the renderer.
>> 
>> I'd imagine both audioElement and videoElement have this "append" call. 
>> 
>> I did some quick tests with trying to keep a video and an audio in sync (for closed audiodescriptions). You have to pay attention around buffering, but once you have sync it works great.
> 
> One quick follow-up on this. How would you indicate to the player that the video and audio elements must be synchronized ? If one stalls you want the other to pause too. Presumably you want to do playback control and receive playback events through just one of them.

One of the elements (the video) would be considered the "master". The audio ("slave") is seeked when they are out of line. In the player framework, only the events / handlers / etc from the video are exposed, so to the outside world it looks like a single video.

You could make certain exceptions to the master/slave rule. For example, when the audio encounters a buffer-underrun, the video is also paused. 

> Or would it be better to treat this case as a single video element, which handles both audio and video and so has separate appendVideo() and appendAudio() calls ?

That might be nicer, in case one has separate audio and video streams (most publishers don't). That way, you don't have to build fragile javascript frameworks.

It does require more work from browser developers though, so not sure if this would be preferred. 

- Jeroen