[foms] What data is needed for adaptive stream switching?

[Chris Pearce]

On 23/11/2010 12:26 p.m., Benjamin M. Schwartz wrote:
> AFAICT, WebM streams may be "variable frame rate" (VFR), i.e. frames may have arbitrary
> duration and be intermittent.

This is correct.

> Suppose your decode pipeline looks like:
>
> network source ->  demux ->  queue1 (encoded frames) ->  decode ->  queue2
> (decoded frames) ->  painter
>
> I favor a solution that either (1) exposes the temporal range currently in
> each queue,

To reduce memory footprint we only keep about decoded 10 video frames in 
advance of the playback position (but about 2s of audio). We don't demux 
the entirety of our downloaded data, also to save memory.

> or (2) exposes the browser's estimate of the time remaining
> before each queue underruns (possibly +inf).  Option 1 is just like your
> design (using @buffered), but extended to distinguish network and CPU limits.

There are two separate problems here, detecting slow network, and 
detecting slow CPU. My @buffered proposal can only detect slow network.

So you want us to expose an attribute which simply reports whether the 
we expect to drop frames in future?

I think predicting frame dropping due to slow CPU in a timely manner is 
hard. Given that we only keep 10 decoded frames queued, even if we post 
an event notification when things go bad, you wouldn't have enough time 
to setup a new connection to switch to a lower quality stream before you 
start to drop frames anyway.

>> Or we could also expose an attribute which simply reports whether the
>> browser is dropping frames, and so you can use this to determine whether
>> you need to switch down to a lower quality stream in order to get smooth
>> playback.
> Please don't make dropped frames the primary metric.  A correctly
> functioning player should adapt so that frames are never dropped.

So you mean guess in advance what resolution and bitrate the user's 
hardware will be able to handle playing? How do you tell the bitrate of 
a VBR video in advance? How do you factor in the affect of variable 
CPU-load JavaScript running in other tabs? Or other videos playing?

>> Given the painting architecture in Firefox, it's very inconvenient to
>> separately count the number of frames which were not painted due to the
>> video being hidden/off screen (this was suggested by Andy Berkheimer).
>> If you wanted to detect this, you could detect when the "painted frames"
>> counter completely stops incrementing. Even when we're CPU starved, we
>> should be able to paint the occasional frame...
> My impression is that in FF4 and other upcoming browsers, sites doing
> basic video playback use almost no CPU for painting, thanks to GPU
> acceleration.  As such, if the system is CPU-starved it will be due to
> decoding.

No. The browser can be running heavy JavaScript, in multiple tabs. 
People do crazy things with web browsers these days. Decoding is not 
only heavy user of the CPU!

> Also, switching to a stream with lower bitrate or resolution
> may not affect the CPU load from painting.

Sometimes we have to fallback to scaling in the CPU, for example if the 
user's graphics card is blacklisted due to buggy drivers. :(

On a related note, a lower quality stream will be quicker to decode as 
well, so frames will be less likely to be late. This is particularly 
important on mobile platforms, or on underpowered netbooks.

> Of course,, you cannot safely hard-drop frames at the decode stage.  The
> system must decode every frame, then skip ahead when a keyframe arrives.*

If Firefox thinks it's going to audio underrun, we skip the video decode 
up to the next key frame, and resume decoding from there. How else can 
you save time during the decode? We drop frames which are presented late 
as well.

> Due to variance in decode times, it's likely that at the edge of
> CPU-starvation you will see some frames miss their display deadline even
> though the stream can be decoded in real-time on average.  If you can't
> afford len(queue2)>1, then you can substitute deadline-miss statistics to
> indicate when the system is at the edge of CPU starvation, without hard
> frame-dropping.  (You can even avoid frame-dropping altogether by
> displaying those frames even if they show up a few ms late, or by setting
> their deadline a few ms ahead of when they are actually needed.)
>

What would the deadline-miss statistics look like?

We drop frames for two reasons:

   1. It looks like we're going to audio underrun, so we need to shed
      work to keep the audio playing smoothly.
   2. The frame arrived late for painting (possibly due to slow decode,
      or delays in the pipeline).

In case 1 what would you report for the deadline-miss?


Chris P.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.annodex.net/cgi-bin/mailman/private/foms/attachments/20101123/0207a67e/attachment.htm