[foms] Chunked/adaptative streaming at Dailymotion

Fri Nov 5 02:19:56 PDT 2010

> Well, not really. Akamai is sized to support such load for instance.
If you anticipate 120,000 titles, averaging 1 hour in duration, and encode at 10 bitrates (perhaps 100, 250, 350, 500, 1000, 1350, 1750, 2650, 3600, 4800), then each title
consumes ~7.5GB of space.  The library will consume ~900 peta bytes of storage, across 1,200,000 video files.  If the files are physically chunked at 2 seconds, then you
would have ~2,160,000,000 chunk files.

I think it's about 840TB, not 900PB ;-)

If the system requires mutliplexed A/V streams (ala HTTP Live Streaming), and there is a need for multiple audio bitrates, and you expect a few language tracks per title,
then every video stream will be replicated n times (where n is the number of audio bitrates + the # of alternate audio tracks).  We encode 3 audio bitrates, and I will
assume an avg of 2 alternate audio tracks per title (each alternate track is encoded at all bitrates, but not all streams have alternate tracks).  Thus, for this exercise,
n == 5.  Using physically chunked, muxed streams, the footprint is 4 exabytes across 12,000,000,000 files, 4/5 of which is consists of duplicated video bits.  Efficiently
storing and caching these bits might challenge a CDN. Thus, we prefer unmuxed streams with virtual chunking (range requests)

All video streams and audio streams can be muxed and kept in sync in the same MP4 or MKV container (it's the purpose of having multi-tracks-aware containers).
Even different bitrates of audio and video can be stored within the same ISO container.

That said, you're absolutely right: it makes sense to remux/resync at the client side when you have that type of cardinality in combinations. It's just that sometimes you
can't control the client side (Flash, HTML5 video tags as of today, most IPTV STB and connected TV sets (again today), ...), and you need to deliver a pre-muxed valid
fragment.

> My guess is that you're using MPEG2-TS files ...
Our original streaming clients use ASF.  BD-Live and Apple clients use M2TS.  All others are using fragmented ISO.
FYI, our streaming traffic is ~20% of peak US internet traffic, and our content is delivered at the highest bitrate a given client's bandwidth will allow (we currently max out at 4.8mbps).

Yes, we all know about your evil plan to crunch the Internet down : http://www.slate.com/id/2273314 (just kidding) ^_^

Pierre-Yves

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.annodex.net/cgi-bin/mailman/private/foms/attachments/20101105/1253c7b5/attachment.htm 