[Icecast-dev] listen backlog patch

["Thomas B. Rücker"]

Hi,

On 02/19/2015 03:40 PM, Stephan Leemburg wrote:
> I don't know if you like top or bottom quoting. That seems to be a
> big-little endian thing ;-)
>
> So, I will top quote and inline quote.

playing it safe, ha!

> Please see my comments inline, below.
>
> Kind regards,
> Stephan
>
>
> On 02/19/2015 04:18 PM, "Thomas B. Rücker" wrote:
>> Hi,
>>
>> On 02/19/2015 03:07 PM, Stephan Leemburg wrote:
>>> Hello Icecast-dev,
>>>
>>> I am new to this list.
>> Welcome!
> Thank you.
>
>>> I am working for the NPO, the Dutch Public Broadcasting agency.
>>> We do a lot of icecast streaming. We run at least 20 icecast server
>>> instances on our media streaming cluster.
>> That's very nice to hear.
>>
>>> We ran into an issue that clients which where connecting to our streams
>>> seemed to be 'hanging' on the connection setup frequently. The client
>>> 'thinks' it is connected, but no data.
>>>
>>> People suggested that it probably had to do with the poll() call. So, I
>>> looked into that.
>>>
>>> I found that the issue was actually caused by the very low listen
>>> backlog (5).
>>>
>>> On our clusters, we typically set this to 8192. Yes it is high, but we
>>> do a _lot_ of streaming and host very high volume websites.
>> I'm not very familiar with socket programming, so will let Philipp
>> comment on this. Interesting enough this issue hasn't come up so far as
>> far as I can tell and there are some pretty high load deployments out there.
> We often have 'bursts' of new connections. Due to something said on a
> website, radio, tv, some top-2000 end of year event, etc. And when we
> get a lot of simultanious connection requests, this becomes an issue.
>
> I wrote a small server to simulate it. And the linux backlog acts a
> little bit different then expected.

In the context of what you mention just below, yes, I'm immediately
willing to believe that there is a problem. The scale also explains why
it's probably not that commonly encountered.


>> If you can share that info, what sort of concurrent listener load are we
>> talking about?
> Sure I can share, we are a national broadcasting agency funded by tax
> money. So no secrets here ;-)
>
> I just asked the media streaming guys.. 70k icecast connections on a
> regular day and 150k around special broadcasts (like top 2000 around new
> year).

That's rather sizable.
There are two things I'd like to bring up in this context:

* We'd be delighted if you/NPO could share a few things that you learned
are important to deploy Icecast at that scale. Raising the ulimit is
rather obvious, but you might have run into other things.

* Also if there are more issues, we'd like to hear them, as we want to
make Icecast even better.

I personally believe in the simplicity of HTTP streaming.
I've looked at HLS/DASH and there are various issues that make it
unnecessarily hard to deal with for little value in return.

Especially in web browsers the <audio> element starts to shape up and
support simple HTTP streams rather well. The issue of supported codecs
remains though, with Opus being a decent long run candidate.

The main focus for Icecast in this context would be to help achieve
better listener support on mobile devices, which traditionally seem to
favour HLS. Large stations/networks just create an app that wraps things
nicely, but there are many smaller ones. I see VLC as a good candidate
in this context, but for that we also need to improve our stream
directory at http://dir.xiph.org, as it currently only exports one big
XML file. We're hoping to work out a good and flexible JSON API and help
player-software projects integrate it. It's also part of our GSoC ideas.


>>> Currently we are using icecast 2.3. We are migrating to 2.4.
>>> So, I have written patches for 2.3 and 2.4, but also for the current 2.5
>>> git tree.
>>>
>>> Unfortunately, I am a newby when it comes to git (sorry). But I do have
>>> unified diff patch files for the 2.3, 2.4 and 2.5 source trees.
>>>
>>> The patched 2.4 icecast was tested by our media streaming team and they
>>> confirmed that their issue was solved by it.
>>>
>>> Can I submit them (and how)?
>> Just send them as attachments to this list, or open a ticket over at
>> https://trac.xiph.org.
>> If trac is naughty and thinks you're a spammer, please let me know.
>>
>> If there are differences between them, then please the 2.5 and 2.4
>> patches. Otherwise 2.5 will do just fine.
>>
> I have attached the 2.4 and 2.5 patches.

Thanks, Philipp took a look and we see the problem. Addressing this in a
way that doesn't open us by default to DoS will be the main challenge. A
likely outcome would be exposing the value in the configuration and only
documenting it in the "very advanced" part.

I'd expect that we'll address this one way or the other soon and it will
be part of 2.5.0 later this year.


>> Thanks a lot for taking the time to reach out to us!
> Thank you for your Open Source contributions and efforts!
> And as it is Open Source, we can fix problems ourself and share.

As said we deeply appreciate that, it's what makes open source software
better for everyone.


Cheers

Thomas