<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 14 October 2014 15:01, "Thomas B. Rücker" <span dir="ltr"><<a href="mailto:thomas@ruecker.fi" target="_blank">thomas@ruecker.fi</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 10/14/2014 12:56 PM, Matt Gray wrote:<br>
> Hi Thomas,<br>
><br>
> We need a load balancer to allow for a single URL "entry point" to<br>
> multiple Icecast servers - we intend to host multiple Icecast servers<br>
> behind the load balancer so that we can expand / contract capacity as<br>
> required - we would add servers into the load balancer pool. This also<br>
> allows us to scale beyond the number of listeners a single Icecast<br>
> server can support, while still maintaining the "illusion" of a single<br>
> streaming server.<br>
<br>
</span>That is only really relevant if you expect to saturate the upstream<br>
bandwidth of the instance running Icecast. Which nowadays is in the<br>
10-20k listener range at the very least. Clustering through DNS also<br>
removes that limitation.<br></blockquote><div><br></div><div>Unfortunately this is not the case in EC2, as the upstream bandwidth is not as high as one would expect hosting one's own server (the AWS decision is out of my hands unfortunately). With an m3.medium we are able to support about 2000 listeners per server.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
For context, such listener numbers are only achieved by the largest<br>
on-line radio stations.<br>
Does an ELB have more upstream bandwidth than a single instance? Would<br>
be one question. If not (iow >>1Gbit) you need a different approach anyway.<br></blockquote><div><br></div><div>ELB has much greater upstream bandwidth than an of the amazon instance sizes. It's effectively unlimited, it grows as connections increase.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> It also provides redundancy / failover capability by allowing us to<br>
> balance connections across Icecast servers in multiple AWS<br>
> availability zones.<br>
<br>
</span>Redundancy/failover might be worth looking at. I'm not exactly sure if<br>
adding another SPOF is the right approach, but thankfully it's not my<br>
decision.<br>
Out of curiosity, is ELB able to span several AZ?<br></blockquote><div><br></div><div>ELB is not a SPOF (at least an entire Amazon region has to go down for an ELB to completely fail), because it scales to cope with increased connections. ELB is able to span AZs within a region. </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Anyway, Icecast on the listener side is plain HTTP, I'd try that first.<br>
I'm not sure how well it will cope with the extremely long lived<br>
connections though. In my experience adding the complexity of a reverse<br>
proxy decreases Icecast availability and reliability rather than improve it.<br>
<span class=""><br>
<br>
> I'm aware of some "playlist based" loadbalancing techniques using .m3u<br>
> or HTTP redirects to the icecast servers, but this is not ideal for<br>
> us, as a) we need to support some legacy players (eg. set top boxes)<br>
> that don't support redirects or m3u's, and<br>
<br>
</span>I'd not go for a playlist as it relies on clients. I'd probably just use<br>
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about<br>
distribution. One stable and fully transparent URL. That's also easier<br>
to implement for reliable and robust availability in my opinion. Turns<br>
out Amazon even offers it as a service: Amazon Route 53.<br></blockquote><div><br></div><div>Yes we've looked at Route 53, perhaps this is an option we will go with.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> b) using ELB allows us to very easily provision backend health checks,<br>
> multi availability zone redundancy, and auto scaling of Icecast servers.<br>
<br>
</span>OK, that sentence ticks a row on my bingo card. ;-)<br>
You're set on using ELB for some reason and I won't argue further with<br>
it. </blockquote><div><br></div><div>We are certainly not set on it, because of the issues that I've explained (which might well be to do with ELB) I'm just trying to to understand the behaviour of Icecast in that environment.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'll just warn that it's probably not ideal in both HTTP and TCP<br>
mode. You should do significant testing to ensure things work as they do<br>
without it in the way.<br>
Why it doesn't close the TCP connection after the original connection is<br>
gone is not clear to me. It might be a bug or a feature of ELB. You<br>
should actually check AWS documentation or contact Amazon support about<br>
that. </blockquote><div><br></div><div>I have also taken this up with AWS support and their stance is that sockets in CLOSE_WAIT are waiting for the backend server to close them, and there is nothing that the ELB can do about that.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">You can chose to ignore it and the system will eventually kill<br>
them, but you should make sure to have enough of head room for<br>
connections available on the Icecast machine (ulimit and server config<br>
limits).<br></blockquote><div><br></div><div>Unfortunately we can't, because of the issues you mention and also we need to use the disconnection event to log listener usage etc.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I know I sound quite defensive about this topic, but the point I'm<br>
trying to make is, that it adds not well known behaviour, as it's not<br>
fully transparent, and constitutes another single point of failure. Also<br>
there are suitable other options. (I'd even make the same point for a<br>
plain web server)</blockquote><div><br></div><div> You've every right to be defensive :) I'm just trying to understand what causes the CLOSE_WAIT states. You are right, there are plenty of other options.</div><div><br></div><div>In general, what are people using to provide a high availability icecast service? </div><div><br></div><div>Thanks for your time putting together such a comprehensive response.</div></div>
</div></div>
<br>
<span style="font-size:7.5pt;font-family:Verdana,sans-serif"><img src="http://www.7digital.com/stores/assets/7digital.com/7d_EmailSignature.gif"><br><br>This
email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No
liability can be accepted for viruses that might be transferred by this
email or any attachment. Any unauthorised copying of this message or
unauthorised distribution and publication of the information contained
herein are prohibited.<br><br></span><font style="font-family:Arial,Helvetica,sans-serif" size="1"><font style="font-family:Arial,Helvetica,sans-serif" size="1">7digital Limited. Registered office:</font><font style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em" size="1"> </font><font face="Arial, Helvetica, sans-serif" size="1">69 Wilson Street, London EC2A 2BB</font><span style="font-family:Arial,Helvetica,sans-serif;font-size:x-small">.<br>Registered in</span><font size="1"> England and Wales. Registered No. 04843573.</font></font>