[foms] What data is needed for adaptive stream switching?

[Benjamin M. Schwartz]

On 11/22/2010 05:30 PM, Chris Pearce wrote:
> What data does the browser need to expose to javascript to enable 
> clients to do adaptive stream switching?

Overall, my best answer is to have someone look closely at GStreamer's QoS
system.  It is designed to solve this class of problems... and many
browsers might end up using it to implement whatever solution is selected.

> Browsers can expose counters for frames demuxed, decoded and painted to 
> enable detection of cases 4-6.

You appear to be assuming a constant framerate stream.  AFAICT, WebM
streams may be "variable frame rate" (VFR), i.e. frames may have arbitrary
duration and be intermittent.  Highly variable framerates are especially
useful for slideshows.

Suppose your decode pipeline looks like:

network source -> demux -> queue1 (encoded frames) -> decode -> queue2
(decoded frames) -> painter

I favor a solution that either (1) exposes the temporal range currently in
each queue, or (2) exposes the browser's estimate of the time remaining
before each queue underruns (possibly +inf).  Option 1 is just like your
design (using @buffered), but extended to distinguish network and CPU limits.

> Or we could also expose an attribute which simply reports whether the 
> browser is dropping frames, and so you can use this to determine whether 
> you need to switch down to a lower quality stream in order to get smooth 
> playback.

Please don't make dropped frames the primary metric.  A correctly
functioning player should adapt so that frames are never dropped.

> Given the painting architecture in Firefox, it's very inconvenient to 
> separately count the number of frames which were not painted due to the 
> video being hidden/off screen (this was suggested by Andy Berkheimer). 
> If you wanted to detect this, you could detect when the "painted frames" 
> counter completely stops incrementing. Even when we're CPU starved, we 
> should be able to paint the occasional frame...

My impression is that in FF4 and other upcoming browsers, sites doing
basic video playback use almost no CPU for painting, thanks to GPU
acceleration.  As such, if the system is CPU-starved it will be due to
decoding.  Also, switching to a stream with lower bitrate or resolution
may not affect the CPU load from painting.

Of course,, you cannot safely hard-drop frames at the decode stage.  The
system must decode every frame, then skip ahead when a keyframe arrives.*

Due to variance in decode times, it's likely that at the edge of
CPU-starvation you will see some frames miss their display deadline even
though the stream can be decoded in real-time on average.  If you can't
afford len(queue2)>1, then you can substitute deadline-miss statistics to
indicate when the system is at the edge of CPU starvation, without hard
frame-dropping.  (You can even avoid frame-dropping altogether by
displaying those frames even if they show up a few ms late, or by setting
their deadline a few ms ahead of when they are actually needed.)

--Ben

*: For some streams that make heavy use of B-frames, or equivalent
structures in VP8, it is possible to save CPU by dropping these frames,
without corrupting all subsequent video.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
Url : http://lists.annodex.net/cgi-bin/mailman/private/foms/attachments/20101122/940afdec/attachment.pgp