[foms] Proposal: adaptive streaming using open codecs

[Andy Berkheimer]

On Fri, Oct 29, 2010 at 8:59 AM, Jeroen Wijering
<jeroen at longtailvideo.com> wrote:
>
> On Oct 28, 2010, at 1:02 AM, Andy Berkheimer wrote:
>
>>>> * *droppedFrames*: The total number of frames dropped for this playback session (read-only).
>>>> * *decodedFrames*: The total number of frames decoded for this playback session (read-only).
>>>>
>>>> Yep, what Firefox has. Is this the metric you prefer? MY guess is that you'd be more interested in the performance around now (a window of X seconds) than globally, especially when the video stream has switched from low to high quality or vice versa.
>>>
>>> This metric is more "raw", so there's more to do with it. For example, I could setup my own sliding window and calculate the droppedFPS with it, e.g. for blacklisting heuristics with longer half-life times. I could also ignore sudden spikes in the total number of droppedFrames (probably CPU was shortly working on something else), etc.
>>>
>>> In most cases, both droppedFPS and currentFPS would be enough (compare; then blacklist if more than X% is dropped), so we could use this instead. Chris Double then has to update his patch though [1] ...
>>
>> One issue we've run into is that there are often two very different reasons for a video frame to be dropped in a client pipeline:
>> 1) shedding work to maintain timeliness of video rendering
>> 2) optimization when the video window is completely obscured or hidden - in a background tab, etc.
>>
>> Differentiating these cases is important for any adaptive heuristics as well as monitoring and understanding aggregate client performance behavior.  If #2 counts as a droppedFrame then quality problems are over-reported and switching is likely to be too aggressive, if #2 is not counted as a droppedFrame then problems are underreported and switching is likely to be too conservative.  Which calls out for a third counter - hiddenFrames?
>
> Isn't there a way to detect this in javascript, like in ationscript with Event.ACTIVATE? Next, the frames that get dropped b/c the video is not in focus are not counted.

There's window.onfocus() and window.onblur() but they aren't quite
right since they deal with input focus, not visibility.  Demo page:
http://www.thefutureoftheweb.com/demo/2007-05-16-detect-browser-window-focus/

> Perhaps even better to pause video when it's in a background tab. Moreover, perhaps user-agents should enforce this ...

Re: pausing media in background tab.  There are a few use cases where
continuing some media work in a background tab is very useful for the
viewer:
- user really wants to watch high quality video even if they have a
slow connection, pause playback but continue buffering.
- audio-only playback.


-Andy