[Icecast] Running Icecast2 behind Amazon Elastic Load Balancer (ELB)

["Thomas B. Rücker"]

On 10/14/2014 12:56 PM, Matt Gray wrote:
> Hi Thomas,
>
> We need a load balancer to allow for a single URL "entry point" to
> multiple Icecast servers - we intend to host multiple Icecast servers
> behind the load balancer so that we can expand / contract capacity as
> required - we would add servers into the load balancer pool. This also
> allows us to scale beyond the number of listeners a single Icecast
> server can support, while still maintaining the "illusion" of a single
> streaming server.

That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach anyway.


> It also provides redundancy / failover capability by allowing us to
> balance connections across Icecast servers in multiple AWS
> availability zones.

Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.


> I'm aware of some "playlist based" loadbalancing techniques using .m3u
> or HTTP redirects to the icecast servers, but this is not ideal for
> us, as a) we need to support some legacy players (eg. set top boxes)
> that don't support redirects or m3u's, and

I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.


> b) using ELB allows us to very easily provision backend health checks,
> multi availability zone redundancy, and auto scaling of Icecast servers.

OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it. I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that. You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)


Cheers

Thomas