[foms] What data is needed for adaptive stream switching?

[Frank Galligan]

On Tue, Nov 30, 2010 at 5:47 PM, Mark Watson <watsonm at netflix.com> wrote:

>
> On Nov 29, 2010, at 8:05 AM, Frank Galligan wrote:
>
>
>
> On Tue, Nov 23, 2010 at 4:24 PM, Chris Pearce <chris at pearce.org.nz> wrote:
>
>> Thanks for explaining Mark. Much appreciated.
>>
>> On 24/11/2010 6:51 a.m., Mark Watson wrote:
>> > This is where there is scope for experimentation. What I think would be
>> great is to define an API which can indicate these decision points, provide
>> the two data sets (past incoming bandwidth) and (future bitrate of each
>> stream) at some sufficient level of generality and indicate back the
>> decision. Then we can experiment with more and less complex input data and
>> more and less complex decision algorithms.
>>
>> So in terms of what changes to browsers we'd need to make to start
>> experimenting, we'd need to resurrect the @bufferedBytes attribute, add
>> a @currentOffset attribute, and add some way for JS to access the
>> RAP/keyframe index?
>
> I'm not sure what exposing the RAP to the JS for each stream buys you over
> just having a simple switch stream api. Let me back up a bit.
>
> At a high level I think we want the media pipeline to expose information
> about the current presentation to the player through a script interface.
> Then the player can make decisions about what stream the media
> pipeline should be rendering. The decisions made by the player are it's best
> guess so the media pipeline will not stall for any reason. We want to do
> this because trying to define an algorithm that all default players must
> implement is going to be extremely hard. This is my view from reading over
> the posts on this list.
>
> So to me I don't think that a JS player really cares that the media
> pipeline switches to stream N at byte offset X (of course the
> media pipeline does). All the player cares about is that the media pipeline
> switch to stream N as soon as possible. I.E. The player decides that the CPU
> load is too great and wants to switch to a lower resolution as soon as
> possible seamlessly. Or the player decides that the bandwidth looks
> high enough that it would like to switch to a higher bandwidth stream as
> soon as possible seamlessly.
>
> I think having an API that is SwitchTo(index, discontinuous); should be
> good enough for now. (I added a discontinuous parameter in case the player
> wanted to switch and didn't care about it being seamless.) Are there any
> reasons we need to expose RAPs to the player?
>
>
> The problem if you make the information exposed to the JS layer too simple
> is that you end up with essentially only one adaptation algorithm (compare
> incoming bandwidth to stream rates) and then there is no scope for
> experimentation.
>
I understand, and whatever we do now is not set in stone. I think
we will need to iterate a few times before we strike the correct complexity
balance for a spec.

>
> On the other hand, I agree that it could easily be made too complex.
>
> So the question is how "precise" do we expect the adaptation algorithms at
> the JS level to be ? Exposing the switch points is of interest if you expect
> the JS algorithm to be making quite precise decisions based on the current
> amount of received data, remaining distance to the next switch point and
> precise VBR profile of the streams going forward.
>
I think right now most people on this list probably want the JS api to be
precise, so we would have to expose the switch points to the JS api. Just
more complicated for the video tag implementation. I will have to think
about this some more.

>
>
> I can think of a corner case, but I don't think the added complexity to the
> interface and to the player developer is worth it. The case I'm thinking of
> is the player wants to switch to a lower bandwidth stream but stream N which
> the player wants to switch to doesn't have a RAP for 10 seconds while stream
> N-1, which has bandwidth < stream N, has a RAP 2 seconds out. If people felt
> strongly about handling this case we could expose an API like
> SwitchDown(attribute); Then there would also be an API call that can control
> the heuristics if the media pipeline had to switch to another stream with a
> lower value attribute than the desired stream.
>
>
> I'm not sure it's a corner case. Even if I have switch points every 2s if
> my bandwidth drops by 50% then it could be as much as 4s to receive enough
> data to get to the next switch point. I probably need to account for this
> and I may choose an even lower stream rate than otherwise to avoid a stall.
>
True.

>
>
> Maybe we should add the @bufferedBytes data into
>> @buffered, so you can easily map buffered time ranges to byte ranges? I
>> guess these would have to be per-stream if we're playing multiple
>> independent streams.
>>
>> Or would you prefer an explicit download bandwidth and a per stream
>> bitrate measure, calculated by the browser, over a specified time window?
>>
>> In terms of an API which can indicate decision points, maybe an event
>> which fires when playback enters a new chunk? Or fires when the download
>> of a chunk finishes? Be aware we can't guarantee that any DOM events
>> which we fire will arrive in a particularly timely manner, there could
>> be any number of other things going on with the event queue.
>>
> This is another reason that I think a higher level API for the player is
> better.
>
>
>>
>> Is there a direct mapping between a keyframe index and RAP points? What
>> about audio streams in containers which don't index audio? Particularly
>> in the case where we're playing multiple independent streams.
>>
> For WebM I wrote a tool that would create an index file for an audio
> stream.
>
> Frank
>
>
>
>> Chris P.
>>
>> _______________________________________________
>> foms mailing list
>> foms at lists.annodex.net
>> http://lists.annodex.net/cgi-bin/mailman/listinfo/foms
>>
>
> <ATT00001..txt>
>
>
>
> _______________________________________________
> foms mailing list
> foms at lists.annodex.net
> http://lists.annodex.net/cgi-bin/mailman/listinfo/foms
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.annodex.net/cgi-bin/mailman/private/foms/attachments/20101203/0d601bf0/attachment.htm