[foms] Proposal: adaptive streaming using open codecs

[Jeroen Wijering]

On Oct 28, 2010, at 1:02 AM, Andy Berkheimer wrote:

>>> * *droppedFrames*: The total number of frames dropped for this playback session (read-only).
>>> * *decodedFrames*: The total number of frames decoded for this playback session (read-only).
>>> 
>>> Yep, what Firefox has. Is this the metric you prefer? MY guess is that you'd be more interested in the performance around now (a window of X seconds) than globally, especially when the video stream has switched from low to high quality or vice versa.
>> 
>> This metric is more "raw", so there's more to do with it. For example, I could setup my own sliding window and calculate the droppedFPS with it, e.g. for blacklisting heuristics with longer half-life times. I could also ignore sudden spikes in the total number of droppedFrames (probably CPU was shortly working on something else), etc.
>> 
>> In most cases, both droppedFPS and currentFPS would be enough (compare; then blacklist if more than X% is dropped), so we could use this instead. Chris Double then has to update his patch though [1] ...
> 
> One issue we've run into is that there are often two very different reasons for a video frame to be dropped in a client pipeline:
> 1) shedding work to maintain timeliness of video rendering
> 2) optimization when the video window is completely obscured or hidden - in a background tab, etc.
> 
> Differentiating these cases is important for any adaptive heuristics as well as monitoring and understanding aggregate client performance behavior.  If #2 counts as a droppedFrame then quality problems are over-reported and switching is likely to be too aggressive, if #2 is not counted as a droppedFrame then problems are underreported and switching is likely to be too conservative.  Which calls out for a third counter - hiddenFrames?

Isn't there a way to detect this in javascript, like in ationscript with Event.ACTIVATE? Next, the frames that get dropped b/c the video is not in focus are not counted.

Perhaps even better to pause video when it's in a background tab. Moreover, perhaps user-agents should enforce this ...

> Also prefer to have these metrics expose the raw counter, rather than a more nebulously defined instantaneous rate.  More flexibility, as Jeroen points out.

Yes indeed! Also see Mark Watson's comments on preferring "bytesLoaded" over "bandwidth". Same deal.

- Jeroen