[Icecast] Running Icecast2 behind Amazon Elastic Load Balancer (ELB)

[Matt Gray]

On 14 October 2014 15:01, "Thomas B. Rücker" <thomas at ruecker.fi> wrote:

> On 10/14/2014 12:56 PM, Matt Gray wrote:
> > Hi Thomas,
> >
> > We need a load balancer to allow for a single URL "entry point" to
> > multiple Icecast servers - we intend to host multiple Icecast servers
> > behind the load balancer so that we can expand / contract capacity as
> > required - we would add servers into the load balancer pool. This also
> > allows us to scale beyond the number of listeners a single Icecast
> > server can support, while still maintaining the "illusion" of a single
> > streaming server.
>
> That is only really relevant if you expect to saturate the upstream
> bandwidth of the instance running Icecast. Which nowadays is in the
> 10-20k listener range at the very least. Clustering through DNS also
> removes that limitation.
>

Unfortunately this is not the case in EC2, as the upstream bandwidth is not
as high as one would expect hosting one's own server (the AWS decision is
out of my hands unfortunately). With an m3.medium we are able to support
about 2000 listeners per server.


> For context, such listener numbers are only achieved by the largest
> on-line radio stations.
> Does an ELB have more upstream bandwidth than a single instance? Would
> be one question. If not (iow >>1Gbit) you need a different approach anyway.
>

ELB has much greater upstream bandwidth than an of the amazon instance
sizes. It's effectively unlimited, it grows as connections increase.


> > It also provides redundancy / failover capability by allowing us to
> > balance connections across Icecast servers in multiple AWS
> > availability zones.
>
> Redundancy/failover might be worth looking at. I'm not exactly sure if
> adding another SPOF is the right approach, but thankfully it's not my
> decision.
> Out of curiosity, is ELB able to span several AZ?
>

ELB is not a SPOF (at least an entire Amazon region has to go down for an
ELB to completely fail), because it scales to cope with increased
connections. ELB is able to span AZs within a region.

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
> I'm not sure how well it will cope with the extremely long lived
> connections though. In my experience adding the complexity of a reverse
> proxy decreases Icecast availability and reliability rather than improve
> it.
>
>
> > I'm aware of some "playlist based" loadbalancing techniques using .m3u
> > or HTTP redirects to the icecast servers, but this is not ideal for
> > us, as a) we need to support some legacy players (eg. set top boxes)
> > that don't support redirects or m3u's, and
>
> I'd not go for a playlist as it relies on clients. I'd probably just use
> a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
> distribution. One stable and fully transparent URL. That's also easier
> to implement for reliable and robust availability in my opinion. Turns
> out Amazon even offers it as a service: Amazon Route 53.
>

Yes we've looked at Route 53, perhaps this is an option we will go with.

> b) using ELB allows us to very easily provision backend health checks,
> > multi availability zone redundancy, and auto scaling of Icecast servers.
>
> OK, that sentence ticks a row on my bingo card. ;-)
> You're set on using ELB for some reason and I won't argue further with
> it.


We are certainly not set on it, because of the issues that I've explained
(which might well be to do with ELB) I'm just trying to to understand the
behaviour of Icecast in that environment.


> I'll just warn that it's probably not ideal in both HTTP and TCP
> mode. You should do significant testing to ensure things work as they do
> without it in the way.
> Why it doesn't close the TCP connection after the original connection is
> gone is not clear to me. It might be a bug or a feature of ELB. You
> should actually check AWS documentation or contact Amazon support about
> that.


I have also taken this up with AWS support and their stance is that sockets
in CLOSE_WAIT are waiting for the backend server to close them, and there
is nothing that the ELB can do about that.


> You can chose to ignore it and the system will eventually kill
> them, but you should make sure to have enough of head room for
> connections available on the Icecast machine (ulimit and server config
> limits).
>

Unfortunately we can't, because of the issues you mention and also we need
to use the disconnection event to log listener usage etc.


> I know I sound quite defensive about this topic, but the point I'm
> trying to make is, that it adds not well known behaviour, as it's not
> fully transparent, and constitutes another single point of failure. Also
> there are suitable other options. (I'd even make the same point for a
> plain web server)


 You've every right to be defensive :) I'm just trying to understand what
causes the CLOSE_WAIT states. You are right, there are plenty of other
options.

In general, what are people using to provide a high availability icecast
service?

Thanks for your time putting together such a comprehensive response.

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20141014/ab2fbbb2/attachment.htm>