From gvmzsolt at gmail.com  Thu Mar 14 21:27:17 2024
From: gvmzsolt at gmail.com (zsolt makkai)
Date: Thu, 14 Mar 2024 22:27:17 +0100
Subject: [Icecast] Unable to utilize past 1 gbps
Message-ID: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>

Dear Icecast,

We are operating a web radio on a 10 Gbps dedicated server line. The line
bandwidth is tested and available. The web radio is hosted on a Proxmox
virtual environment. We own the physical server itself and made sure to
have allocated the sufficient amount of resources on the virtual machine.

We found that no matter what we do overall upload cannot go over 1 gbps on
1 mount point.
It is not about the maximum number of listeners as a limiting factor but
the overall bandwidth icecast (we are using 2.4.4) is able to utilize.

We can let more listeners to connect only by decreasing the outgoing
bitrate that is going against quality.

We have looked at icecast web stations globally and what we have found is
that NON of them are exceeding the 1 gbps limit. Like servers having
maximum number of listeners (18000) but using only 48 kbps bitrate which is
around 900 mbps...

Please advice!

Is this really the limit that one icecast server can utilize/mountpoint?

Please link a server that goes over the 1 gbps limit continuously...

Thank you for your help!

Best Regards,

Zsolt Makkai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240314/5ea5eb51/attachment.htm>

From marius at flage.org  Thu Mar 14 21:54:04 2024
From: marius at flage.org (Marius Flage)
Date: Thu, 14 Mar 2024 22:54:04 +0100
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
Message-ID: <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>

I don't know if any such limitation exists, but maxing out just shy of 1Gbps sounds a bit more like an interface's line speed limited/set to 1Gbps somewhere in your production chain? You wrote that you have tested the bw - what did you use? Iperf? Speedtest?Maybe consider scaling with additional Icecast servers and use a load balancer in front? And then just scale accordingly? This sounds like an interesting challenge. Which radio station is this, if I may ask?--MariusSendt fra min Galaxy
-------- Opprinnelig melding --------Fra: zsolt makkai <gvmzsolt at gmail.com> Dato: 14.03.2024  22:28  (GMT+01:00) Til: icecast at xiph.org Emne: [Icecast] Unable to utilize past 1 gbps Dear Icecast,We are operating a web radio on a 10 Gbps dedicated server line. The line bandwidth?is tested and available. The web radio is hosted on a Proxmox virtual environment. We own the physical server itself and made sure to have allocated the sufficient amount of resources?on the virtual machine.We found that no matter what we do overall upload cannot go over 1 gbps on 1 mount point.It is not about the maximum number of listeners as a limiting?factor but the overall bandwidth icecast (we are using 2.4.4) is able to utilize.?We can let more listeners to connect only by decreasing the outgoing bitrate that?is?going against quality.We have looked at icecast web stations globally and what we have found is that NON of them are exceeding the 1 gbps limit. Like servers having maximum number of listeners (18000) but using only 48 kbps bitrate?which is around 900 mbps...Please advice!?Is this really the limit that one icecast server can utilize/mountpoint?Please link a server that goes over the 1 gbps limit continuously...Thank you for your help!Best Regards,Zsolt Makkai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240314/885c89e0/attachment.htm>

From gvmzsolt at gmail.com  Thu Mar 14 22:39:50 2024
From: gvmzsolt at gmail.com (zsolt makkai)
Date: Thu, 14 Mar 2024 23:39:50 +0100
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
References: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
 <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
Message-ID: <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>

Dear Marius,

Our station is Megadanceradio in Hungary.

Our status page for the master server is:
http://45.67.158.93:8000/status.xsl  For the slave:
http://45.67.158.94:8000/status.xsl

We are peaking at around 11.00 am and 13.00 pm in the afternoon.

We have tested extensively the bandwith with speedtest. We tried with iperf
but that did not finish for a long time so we stopped it.

Currently we are load balancing on two 10 gbps dedicated lines. So for now
we are ok, but for the future we do not know for how long it will last.
Currently we are limiting the number of listeners to 5200 on each server so
as not to cripple the system. When we are approaching the bandwidth limit
we are getting loads of timeouts until finally not being able to access
even the status page.

That was our second idea that somewhere in the production chain there is a
limit set but we did not find anything.

Thank you for your help!

Best,

Zsolt

Marius Flage <marius at flage.org> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
23:01):

> I don't know if any such limitation exists, but maxing out just shy of
> 1Gbps sounds a bit more like an interface's line speed limited/set to 1Gbps
> somewhere in your production chain? You wrote that you have tested the bw -
> what did you use? Iperf? Speedtest?
>
> Maybe consider scaling with additional Icecast servers and use a load
> balancer in front? And then just scale accordingly? This sounds like an
> interesting challenge. Which radio station is this, if I may ask?
>
> --
> Marius
>
>
> Sendt fra min Galaxy
>
>
> -------- Opprinnelig melding --------
> Fra: zsolt makkai <gvmzsolt at gmail.com>
> Dato: 14.03.2024 22:28 (GMT+01:00)
> Til: icecast at xiph.org
> Emne: [Icecast] Unable to utilize past 1 gbps
>
> Dear Icecast,
>
> We are operating a web radio on a 10 Gbps dedicated server line. The line
> bandwidth is tested and available. The web radio is hosted on a Proxmox
> virtual environment. We own the physical server itself and made sure to
> have allocated the sufficient amount of resources on the virtual machine.
>
> We found that no matter what we do overall upload cannot go over 1 gbps on
> 1 mount point.
> It is not about the maximum number of listeners as a limiting factor but
> the overall bandwidth icecast (we are using 2.4.4) is able to utilize.
>
> We can let more listeners to connect only by decreasing the outgoing
> bitrate that is going against quality.
>
> We have looked at icecast web stations globally and what we have found is
> that NON of them are exceeding the 1 gbps limit. Like servers having
> maximum number of listeners (18000) but using only 48 kbps bitrate which is
> around 900 mbps...
>
> Please advice!
>
> Is this really the limit that one icecast server can utilize/mountpoint?
>
> Please link a server that goes over the 1 gbps limit continuously...
>
> Thank you for your help!
>
> Best Regards,
>
> Zsolt Makkai
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240314/2eca3bea/attachment.htm>

From gvmzsolt at gmail.com  Thu Mar 14 22:52:32 2024
From: gvmzsolt at gmail.com (zsolt makkai)
Date: Thu, 14 Mar 2024 23:52:32 +0100
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>
References: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
 <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
 <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>
Message-ID: <CAMMX_xJCYiDpxSyMB7fR-0_vgovPBDWguM5_zZ7YLR7CtgBEcA@mail.gmail.com>

Dear Marius,

In addition to that attached please find a screenshot from Proxmox on
statistics.

Do you know any ice cast servers that puts out more than 1 gbps for a
longer period? Just curious whether anyone was able to go beyond 1 gbps for
an extensive time.

Thank you!

Best,

Zsolt


zsolt makkai <gvmzsolt at gmail.com> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
23:39):

> Dear Marius,
>
> Our station is Megadanceradio in Hungary.
>
> Our status page for the master server is:
> http://45.67.158.93:8000/status.xsl  For the slave:
> http://45.67.158.94:8000/status.xsl
>
> We are peaking at around 11.00 am and 13.00 pm in the afternoon.
>
> We have tested extensively the bandwith with speedtest. We tried with
> iperf but that did not finish for a long time so we stopped it.
>
> Currently we are load balancing on two 10 gbps dedicated lines. So for now
> we are ok, but for the future we do not know for how long it will last.
> Currently we are limiting the number of listeners to 5200 on each server so
> as not to cripple the system. When we are approaching the bandwidth limit
> we are getting loads of timeouts until finally not being able to access
> even the status page.
>
> That was our second idea that somewhere in the production chain there is a
> limit set but we did not find anything.
>
> Thank you for your help!
>
> Best,
>
> Zsolt
>
> Marius Flage <marius at flage.org> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
> 23:01):
>
>> I don't know if any such limitation exists, but maxing out just shy of
>> 1Gbps sounds a bit more like an interface's line speed limited/set to 1Gbps
>> somewhere in your production chain? You wrote that you have tested the bw -
>> what did you use? Iperf? Speedtest?
>>
>> Maybe consider scaling with additional Icecast servers and use a load
>> balancer in front? And then just scale accordingly? This sounds like an
>> interesting challenge. Which radio station is this, if I may ask?
>>
>> --
>> Marius
>>
>>
>> Sendt fra min Galaxy
>>
>>
>> -------- Opprinnelig melding --------
>> Fra: zsolt makkai <gvmzsolt at gmail.com>
>> Dato: 14.03.2024 22:28 (GMT+01:00)
>> Til: icecast at xiph.org
>> Emne: [Icecast] Unable to utilize past 1 gbps
>>
>> Dear Icecast,
>>
>> We are operating a web radio on a 10 Gbps dedicated server line. The line
>> bandwidth is tested and available. The web radio is hosted on a Proxmox
>> virtual environment. We own the physical server itself and made sure to
>> have allocated the sufficient amount of resources on the virtual machine.
>>
>> We found that no matter what we do overall upload cannot go over 1 gbps
>> on 1 mount point.
>> It is not about the maximum number of listeners as a limiting factor but
>> the overall bandwidth icecast (we are using 2.4.4) is able to utilize.
>>
>> We can let more listeners to connect only by decreasing the outgoing
>> bitrate that is going against quality.
>>
>> We have looked at icecast web stations globally and what we have found is
>> that NON of them are exceeding the 1 gbps limit. Like servers having
>> maximum number of listeners (18000) but using only 48 kbps bitrate which is
>> around 900 mbps...
>>
>> Please advice!
>>
>> Is this really the limit that one icecast server can utilize/mountpoint?
>>
>> Please link a server that goes over the 1 gbps limit continuously...
>>
>> Thank you for your help!
>>
>> Best Regards,
>>
>> Zsolt Makkai
>> _______________________________________________
>> Icecast mailing list
>> Icecast at xiph.org
>> http://lists.xiph.org/mailman/listinfo/icecast
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240314/377b3e1c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Proxmox server 1.JPG
Type: image/jpeg
Size: 212510 bytes
Desc: not available
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240314/377b3e1c/attachment.jpe>

From marius at flage.org  Fri Mar 15 19:24:28 2024
From: marius at flage.org (Marius Flage)
Date: Fri, 15 Mar 2024 20:24:28 +0100
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <CAMMX_xJCYiDpxSyMB7fR-0_vgovPBDWguM5_zZ7YLR7CtgBEcA@mail.gmail.com>
References: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
 <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
 <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>
 <CAMMX_xJCYiDpxSyMB7fR-0_vgovPBDWguM5_zZ7YLR7CtgBEcA@mail.gmail.com>
Message-ID: <72bb6dd4-09e3-4f85-9113-d272ad8a2093@flage.org>

Hi Zsolt,

looking at the metrics from Proxmox it doesn't look like anything breaks 
there. What about at the operating system level? Could there be some 
limits there? Have you checked corresponding kernel or system logs? 
Which OS/distribution are you using? Checked the logs there? If nothing 
screams there, maybe start looking at a load balancer and scale with 
several more VMs?

--
Marius

On 14.03.2024 23:52, zsolt makkai wrote:
> Dear Marius,
>
> In addition to that attached please find a screenshot from Proxmox on 
> statistics.
>
> Do you know any ice cast servers that puts out more than 1 gbps for a 
> longer period? Just curious whether anyone was able to go beyond 1 
> gbps for an extensive time.
>
> Thank you!
>
> Best,
>
> Zsolt
>
>
> zsolt makkai <gvmzsolt at gmail.com> ezt ?rta (id?pont: 2024. m?rc. 14., 
> Cs, 23:39):
>
>     Dear Marius,
>
>     Our station is Megadanceradio in Hungary.
>
>     Our status page for the master server is:
>     http://45.67.158.93:8000/status.xsl For the slave:
>     http://45.67.158.94:8000/status.xsl
>
>     We are peaking at around 11.00 am and 13.00 pm in the afternoon.
>
>     We have tested extensively the bandwith with speedtest. We tried
>     with iperf but that did not finish for a long time so we stopped it.
>
>     Currently we are load balancing on two 10 gbps dedicated lines. So
>     for now we are ok, but for the future we do not know for how long
>     it will last. Currently we are limiting the number of listeners to
>     5200 on each server so as not to cripple the system. When we are
>     approaching the bandwidth?limit we are getting loads of timeouts
>     until finally not being able to access even the status page.
>
>     That was our second idea that somewhere in the production chain
>     there is a limit set but we did not find anything.
>
>     Thank you for your help!
>
>     Best,
>
>     Zsolt
>
>     Marius Flage <marius at flage.org> ezt ?rta (id?pont: 2024. m?rc.
>     14., Cs, 23:01):
>
>         I don't know if any such limitation exists, but maxing out
>         just shy of 1Gbps sounds a bit more like an interface's line
>         speed limited/set to 1Gbps somewhere in your production chain?
>         You wrote that you have tested the bw - what did you use?
>         Iperf? Speedtest?
>
>         Maybe consider scaling with additional Icecast servers and use
>         a load balancer in front? And then just scale accordingly?
>         This sounds like an interesting challenge. Which radio station
>         is this, if I may ask?
>
>         --
>         Marius
>
>
>         Sendt fra min Galaxy
>
>
>         -------- Opprinnelig melding --------
>         Fra: zsolt makkai <gvmzsolt at gmail.com>
>         Dato: 14.03.2024 22:28 (GMT+01:00)
>         Til: icecast at xiph.org
>         Emne: [Icecast] Unable to utilize past 1 gbps
>
>         Dear Icecast,
>
>         We are operating a web radio on a 10 Gbps dedicated server
>         line. The line bandwidth?is tested and available. The web
>         radio is hosted on a Proxmox virtual environment. We own the
>         physical server itself and made sure to have allocated the
>         sufficient amount of resources?on the virtual machine.
>
>         We found that no matter what we do overall upload cannot go
>         over 1 gbps on 1 mount point.
>         It is not about the maximum number of listeners as a
>         limiting?factor but the overall bandwidth icecast (we are
>         using 2.4.4) is able to utilize.
>
>         We can let more listeners to connect only by decreasing the
>         outgoing bitrate that?is?going against quality.
>
>         We have looked at icecast web stations globally and what we
>         have found is that NON of them are exceeding the 1 gbps limit.
>         Like servers having maximum number of listeners (18000) but
>         using only 48 kbps bitrate?which is around 900 mbps...
>
>         Please advice!
>
>         Is this really the limit that one icecast server can
>         utilize/mountpoint?
>
>         Please link a server that goes over the 1 gbps limit
>         continuously...
>
>         Thank you for your help!
>
>         Best Regards,
>
>         Zsolt Makkai
>         _______________________________________________
>         Icecast mailing list
>         Icecast at xiph.org
>         http://lists.xiph.org/mailman/listinfo/icecast
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240315/34461b64/attachment.htm>

From bigwavedave at gmail.com  Fri Mar 15 20:24:27 2024
From: bigwavedave at gmail.com (Dave Breiland)
Date: Fri, 15 Mar 2024 13:24:27 -0700
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <72bb6dd4-09e3-4f85-9113-d272ad8a2093@flage.org>
References: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
 <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
 <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>
 <CAMMX_xJCYiDpxSyMB7fR-0_vgovPBDWguM5_zZ7YLR7CtgBEcA@mail.gmail.com>
 <72bb6dd4-09e3-4f85-9113-d272ad8a2093@flage.org>
Message-ID: <CAJFG59GsN1_eijBDyTXG4YRehecv_G1Apm=v++ORUA79neV6mQ@mail.gmail.com>

A quick Google search indicates that it might be a Proxmox limitation:
https://forum.proxmox.com/threads/i-cant-exceed-1gb-of-internet-with-a-10gb-network-card.128710/



On Fri, Mar 15, 2024 at 12:24?PM Marius Flage <marius at flage.org> wrote:

> Hi Zsolt,
>
> looking at the metrics from Proxmox it doesn't look like anything breaks
> there. What about at the operating system level? Could there be some limits
> there? Have you checked corresponding kernel or system logs? Which
> OS/distribution are you using? Checked the logs there? If nothing screams
> there, maybe start looking at a load balancer and scale with several more
> VMs?
>
> --
> Marius
> On 14.03.2024 23:52, zsolt makkai wrote:
>
> Dear Marius,
>
> In addition to that attached please find a screenshot from Proxmox on
> statistics.
>
> Do you know any ice cast servers that puts out more than 1 gbps for a
> longer period? Just curious whether anyone was able to go beyond 1 gbps for
> an extensive time.
>
> Thank you!
>
> Best,
>
> Zsolt
>
>
> zsolt makkai <gvmzsolt at gmail.com> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
> 23:39):
>
>> Dear Marius,
>>
>> Our station is Megadanceradio in Hungary.
>>
>> Our status page for the master server is:
>> http://45.67.158.93:8000/status.xsl  For the slave:
>> http://45.67.158.94:8000/status.xsl
>>
>> We are peaking at around 11.00 am and 13.00 pm in the afternoon.
>>
>> We have tested extensively the bandwith with speedtest. We tried with
>> iperf but that did not finish for a long time so we stopped it.
>>
>> Currently we are load balancing on two 10 gbps dedicated lines. So for
>> now we are ok, but for the future we do not know for how long it will last.
>> Currently we are limiting the number of listeners to 5200 on each server so
>> as not to cripple the system. When we are approaching the bandwidth limit
>> we are getting loads of timeouts until finally not being able to access
>> even the status page.
>>
>> That was our second idea that somewhere in the production chain there is
>> a limit set but we did not find anything.
>>
>> Thank you for your help!
>>
>> Best,
>>
>> Zsolt
>>
>> Marius Flage <marius at flage.org> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
>> 23:01):
>>
>>> I don't know if any such limitation exists, but maxing out just shy of
>>> 1Gbps sounds a bit more like an interface's line speed limited/set to 1Gbps
>>> somewhere in your production chain? You wrote that you have tested the bw -
>>> what did you use? Iperf? Speedtest?
>>>
>>> Maybe consider scaling with additional Icecast servers and use a load
>>> balancer in front? And then just scale accordingly? This sounds like an
>>> interesting challenge. Which radio station is this, if I may ask?
>>>
>>> --
>>> Marius
>>>
>>>
>>> Sendt fra min Galaxy
>>>
>>>
>>> -------- Opprinnelig melding --------
>>> Fra: zsolt makkai <gvmzsolt at gmail.com>
>>> Dato: 14.03.2024 22:28 (GMT+01:00)
>>> Til: icecast at xiph.org
>>> Emne: [Icecast] Unable to utilize past 1 gbps
>>>
>>> Dear Icecast,
>>>
>>> We are operating a web radio on a 10 Gbps dedicated server line. The
>>> line bandwidth is tested and available. The web radio is hosted on a
>>> Proxmox virtual environment. We own the physical server itself and made
>>> sure to have allocated the sufficient amount of resources on the virtual
>>> machine.
>>>
>>> We found that no matter what we do overall upload cannot go over 1 gbps
>>> on 1 mount point.
>>> It is not about the maximum number of listeners as a limiting factor but
>>> the overall bandwidth icecast (we are using 2.4.4) is able to utilize.
>>>
>>> We can let more listeners to connect only by decreasing the outgoing
>>> bitrate that is going against quality.
>>>
>>> We have looked at icecast web stations globally and what we have found
>>> is that NON of them are exceeding the 1 gbps limit. Like servers having
>>> maximum number of listeners (18000) but using only 48 kbps bitrate which is
>>> around 900 mbps...
>>>
>>> Please advice!
>>>
>>> Is this really the limit that one icecast server can utilize/mountpoint?
>>>
>>> Please link a server that goes over the 1 gbps limit continuously...
>>>
>>> Thank you for your help!
>>>
>>> Best Regards,
>>>
>>> Zsolt Makkai
>>> _______________________________________________
>>> Icecast mailing list
>>> Icecast at xiph.org
>>> http://lists.xiph.org/mailman/listinfo/icecast
>>>
>>
> _______________________________________________
> Icecast mailing listIcecast at xiph.orghttp://lists.xiph.org/mailman/listinfo/icecast
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240315/7d3c76e5/attachment.htm>

From bigwavedave at gmail.com  Fri Mar 15 20:25:39 2024
From: bigwavedave at gmail.com (Dave Breiland)
Date: Fri, 15 Mar 2024 13:25:39 -0700
Subject: [Icecast] Unable to utilize past 1 gbps
In-Reply-To: <CAJFG59GsN1_eijBDyTXG4YRehecv_G1Apm=v++ORUA79neV6mQ@mail.gmail.com>
References: <CAMMX_xLekdxhkR_7zmwwwz_z8kxYMmYOzTa1aFAxNKtK9JgR6A@mail.gmail.com>
 <20240314215409.71E4D32204C@hobbiton.radiomeloy.no>
 <CAMMX_xKX4WrN_=uy2P2ZK0B6p1ZMri8epS0SbQDRNzCo1KoHWg@mail.gmail.com>
 <CAMMX_xJCYiDpxSyMB7fR-0_vgovPBDWguM5_zZ7YLR7CtgBEcA@mail.gmail.com>
 <72bb6dd4-09e3-4f85-9113-d272ad8a2093@flage.org>
 <CAJFG59GsN1_eijBDyTXG4YRehecv_G1Apm=v++ORUA79neV6mQ@mail.gmail.com>
Message-ID: <CAJFG59HexrDbvq3KqjfBaaggZO8yop2M_yPSjkZD2STtp94QBg@mail.gmail.com>

or more accurately... a Proxmox configuration situation that is causing
it.  It seems others have run into this before with other applications.

On Fri, Mar 15, 2024 at 1:24?PM Dave Breiland <bigwavedave at gmail.com> wrote:

> A quick Google search indicates that it might be a Proxmox limitation:
>
> https://forum.proxmox.com/threads/i-cant-exceed-1gb-of-internet-with-a-10gb-network-card.128710/
>
>
>
> On Fri, Mar 15, 2024 at 12:24?PM Marius Flage <marius at flage.org> wrote:
>
>> Hi Zsolt,
>>
>> looking at the metrics from Proxmox it doesn't look like anything breaks
>> there. What about at the operating system level? Could there be some limits
>> there? Have you checked corresponding kernel or system logs? Which
>> OS/distribution are you using? Checked the logs there? If nothing screams
>> there, maybe start looking at a load balancer and scale with several more
>> VMs?
>>
>> --
>> Marius
>> On 14.03.2024 23:52, zsolt makkai wrote:
>>
>> Dear Marius,
>>
>> In addition to that attached please find a screenshot from Proxmox on
>> statistics.
>>
>> Do you know any ice cast servers that puts out more than 1 gbps for a
>> longer period? Just curious whether anyone was able to go beyond 1 gbps for
>> an extensive time.
>>
>> Thank you!
>>
>> Best,
>>
>> Zsolt
>>
>>
>> zsolt makkai <gvmzsolt at gmail.com> ezt ?rta (id?pont: 2024. m?rc. 14.,
>> Cs, 23:39):
>>
>>> Dear Marius,
>>>
>>> Our station is Megadanceradio in Hungary.
>>>
>>> Our status page for the master server is:
>>> http://45.67.158.93:8000/status.xsl  For the slave:
>>> http://45.67.158.94:8000/status.xsl
>>>
>>> We are peaking at around 11.00 am and 13.00 pm in the afternoon.
>>>
>>> We have tested extensively the bandwith with speedtest. We tried with
>>> iperf but that did not finish for a long time so we stopped it.
>>>
>>> Currently we are load balancing on two 10 gbps dedicated lines. So for
>>> now we are ok, but for the future we do not know for how long it will last.
>>> Currently we are limiting the number of listeners to 5200 on each server so
>>> as not to cripple the system. When we are approaching the bandwidth limit
>>> we are getting loads of timeouts until finally not being able to access
>>> even the status page.
>>>
>>> That was our second idea that somewhere in the production chain there is
>>> a limit set but we did not find anything.
>>>
>>> Thank you for your help!
>>>
>>> Best,
>>>
>>> Zsolt
>>>
>>> Marius Flage <marius at flage.org> ezt ?rta (id?pont: 2024. m?rc. 14., Cs,
>>> 23:01):
>>>
>>>> I don't know if any such limitation exists, but maxing out just shy of
>>>> 1Gbps sounds a bit more like an interface's line speed limited/set to 1Gbps
>>>> somewhere in your production chain? You wrote that you have tested the bw -
>>>> what did you use? Iperf? Speedtest?
>>>>
>>>> Maybe consider scaling with additional Icecast servers and use a load
>>>> balancer in front? And then just scale accordingly? This sounds like an
>>>> interesting challenge. Which radio station is this, if I may ask?
>>>>
>>>> --
>>>> Marius
>>>>
>>>>
>>>> Sendt fra min Galaxy
>>>>
>>>>
>>>> -------- Opprinnelig melding --------
>>>> Fra: zsolt makkai <gvmzsolt at gmail.com>
>>>> Dato: 14.03.2024 22:28 (GMT+01:00)
>>>> Til: icecast at xiph.org
>>>> Emne: [Icecast] Unable to utilize past 1 gbps
>>>>
>>>> Dear Icecast,
>>>>
>>>> We are operating a web radio on a 10 Gbps dedicated server line. The
>>>> line bandwidth is tested and available. The web radio is hosted on a
>>>> Proxmox virtual environment. We own the physical server itself and made
>>>> sure to have allocated the sufficient amount of resources on the virtual
>>>> machine.
>>>>
>>>> We found that no matter what we do overall upload cannot go over 1 gbps
>>>> on 1 mount point.
>>>> It is not about the maximum number of listeners as a limiting factor
>>>> but the overall bandwidth icecast (we are using 2.4.4) is able to utilize.
>>>>
>>>> We can let more listeners to connect only by decreasing the outgoing
>>>> bitrate that is going against quality.
>>>>
>>>> We have looked at icecast web stations globally and what we have found
>>>> is that NON of them are exceeding the 1 gbps limit. Like servers having
>>>> maximum number of listeners (18000) but using only 48 kbps bitrate which is
>>>> around 900 mbps...
>>>>
>>>> Please advice!
>>>>
>>>> Is this really the limit that one icecast server can utilize/mountpoint?
>>>>
>>>> Please link a server that goes over the 1 gbps limit continuously...
>>>>
>>>> Thank you for your help!
>>>>
>>>> Best Regards,
>>>>
>>>> Zsolt Makkai
>>>> _______________________________________________
>>>> Icecast mailing list
>>>> Icecast at xiph.org
>>>> http://lists.xiph.org/mailman/listinfo/icecast
>>>>
>>>
>> _______________________________________________
>> Icecast mailing listIcecast at xiph.orghttp://lists.xiph.org/mailman/listinfo/icecast
>>
>> _______________________________________________
>> Icecast mailing list
>> Icecast at xiph.org
>> http://lists.xiph.org/mailman/listinfo/icecast
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240315/d46f42dd/attachment.htm>

From talldarkandstrange at icloud.com  Tue Mar 19 18:41:11 2024
From: talldarkandstrange at icloud.com (TDAS)
Date: Tue, 19 Mar 2024 18:41:11 +0000
Subject: [Icecast] Reverse Proxy
Message-ID: <084BC1D3-A67E-443F-9AB7-FFC5E5FEE43B@icloud.com>

Hi folks

What's the 2024 line of thinking about reverse proxies for icecast?

Not talking massive loads like the 1Gbs discussion I saw the other day (man, how I'd like to have those kind of issues!) :) 

Specifically, Traefik handling https for a few icecast URLs, each with their own docker container? 

Rick

From wayne at cffcs.com  Wed Mar 20 17:16:49 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Wed, 20 Mar 2024 13:16:49 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
Message-ID: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>

Hello everyone.

This is something I am interested in learning more about.
Let's say you have an audience of (what is in the subject line.)
What kind of infrastructure do you need for Icecast?

I asked a similar question about the liquidsoap group.

In Windows and Linux web servers, we can create a forest for our web servers.
Send traffic to different servers to even the workload.

Can we do something like this with the Icecast servers?
(or)
Will we have to install new VMs, add the heavy stations on that one,
and send the new traffic there?

Thanks for any discussion on this.
Wayne


From phschafft at de.loewenfelsen.net  Wed Mar 20 17:55:25 2024
From: phschafft at de.loewenfelsen.net (Philipp Schafft)
Date: Wed, 20 Mar 2024 17:55:25 +0000
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
Message-ID: <325057a1f4da28310f4d589dd86188976348e0bc.camel@de.loewenfelsen.net>

Good morning,

On Wed, 2024-03-20 at 13:16 -0400, Wayne Barron wrote:
> Hello everyone.
> 
> This is something I am interested in learning more about.
> Let's say you have an audience of (what is in the subject line.)
> What kind of infrastructure do you need for Icecast?
> 
> I asked a similar question about the liquidsoap group.
> 
> In Windows and Linux web servers, we can create a forest for our web
> servers. Send traffic to different servers to even the workload.
> 
> Can we do something like this with the Icecast servers?
> (or)
> Will we have to install new VMs, add the heavy stations on that one,
> and send the new traffic there?

Generally you can just run any number of servers in a cluster. Often
with a single or double layer master-slave setup (to also add
redundancy on the source side).


Most of the time the network is the bottle neck. For example for 10k
listeners at 112kbit/s you need something between 1.5Gbit/s and
2.5Gbit/s of network connectivity.

You also want to take into account that at times you want to maintain
your cluster. Or you might also be hit by some bad luck and some part
of your cluster goes down.

You may also want to spread your load geographically.

It's not magic but there is a bit to consider.


A while ago we actually had two presentations on this topic. One on
small setups with less than 30k simultaneous listeners and one for
setups with more.

If there is interest from a few people I would be happy to host an open
discussion or something.

Hope that helps to give you an initial overview.

With best regards,


-- 
Philipp Schafft (CEO/Gesch?ftsf?hrer)
Telephone:???????????+49.3535 490 17 92
Website:?????????????https://www.loewenfelsen.net/
Follow us:???????????https://www.linkedin.com/company/loewenfelsen/
Gesch?ftsf?hrer/CEO: Philipp Schafft

L?wenfelsen UG (haftungsbeschr?nkt)?????Registration number:
Bickinger Stra?e 21?????????????????????HRB 12308 CB
04916 Herzberg (Elster)?????????????????VATIN/USt-ID:
Germany?????????????????????????????????DE305133015


From wayne at cffcs.com  Wed Mar 20 18:23:24 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Wed, 20 Mar 2024 14:23:24 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <325057a1f4da28310f4d589dd86188976348e0bc.camel@de.loewenfelsen.net>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <325057a1f4da28310f4d589dd86188976348e0bc.camel@de.loewenfelsen.net>
Message-ID: <CAJatk5=H+C_xPj1=3n1+BTVng0CQSOiTUTJBtqc=35E5BpKtPg@mail.gmail.com>

Thanks, Phillip.
This is my thread over on Liquidsoap.
https://github.com/savonet/liquidsoap/discussions/3024
This is one of the links that was provided.
https://archive.fosdem.org/2020/schedule/event/om_audio_streaming/
They show where they have.
2 million daily listeners
200,000 concurrent listeners daily.

This is something I am very interested in. I am still working on the
frontend at the moment (The Website part of the Radio Station)
I will keep up with this thread for more information from you and others.

Thanks again, Phillip.
Wayne

On Wed, Mar 20, 2024 at 2:07?PM Philipp Schafft
<phschafft at de.loewenfelsen.net> wrote:
>
> Good morning,
>
> On Wed, 2024-03-20 at 13:16 -0400, Wayne Barron wrote:
> > Hello everyone.
> >
> > This is something I am interested in learning more about.
> > Let's say you have an audience of (what is in the subject line.)
> > What kind of infrastructure do you need for Icecast?
> >
> > I asked a similar question about the liquidsoap group.
> >
> > In Windows and Linux web servers, we can create a forest for our web
> > servers. Send traffic to different servers to even the workload.
> >
> > Can we do something like this with the Icecast servers?
> > (or)
> > Will we have to install new VMs, add the heavy stations on that one,
> > and send the new traffic there?
>
> Generally you can just run any number of servers in a cluster. Often
> with a single or double layer master-slave setup (to also add
> redundancy on the source side).
>
>
> Most of the time the network is the bottle neck. For example for 10k
> listeners at 112kbit/s you need something between 1.5Gbit/s and
> 2.5Gbit/s of network connectivity.
>
> You also want to take into account that at times you want to maintain
> your cluster. Or you might also be hit by some bad luck and some part
> of your cluster goes down.
>
> You may also want to spread your load geographically.
>
> It's not magic but there is a bit to consider.
>
>
> A while ago we actually had two presentations on this topic. One on
> small setups with less than 30k simultaneous listeners and one for
> setups with more.
>
> If there is interest from a few people I would be happy to host an open
> discussion or something.
>
> Hope that helps to give you an initial overview.
>
> With best regards,
>
>
> --
> Philipp Schafft (CEO/Gesch?ftsf?hrer)
> Telephone:           +49.3535 490 17 92
> Website:             https://www.loewenfelsen.net/
> Follow us:           https://www.linkedin.com/company/loewenfelsen/
> Gesch?ftsf?hrer/CEO: Philipp Schafft
>
> L?wenfelsen UG (haftungsbeschr?nkt)     Registration number:
> Bickinger Stra?e 21                     HRB 12308 CB
> 04916 Herzberg (Elster)                 VATIN/USt-ID:
> Germany                                 DE305133015
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast


From fredg at paravelsystems.com  Wed Mar 20 19:53:07 2024
From: fredg at paravelsystems.com (Fred Gleason)
Date: Wed, 20 Mar 2024 15:53:07 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
Message-ID: <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>

On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com> wrote:

> In Windows and Linux web servers, we can create a forest for our web servers.
> Send traffic to different servers to even the workload.
> 
> Can we do something like this with the Icecast servers?
> (or)
> Will we have to install new VMs, add the heavy stations on that one,
> and send the new traffic there?

Ok, I?m going to be ?that guy??

I would argue that, as soon as you?ve hit an audience size of 10,000 or more (especially if that audience is at all geographically dispersed), IceCast is basically off the table. The reason why can be summarized in three letters: ?CDN? [Content Distribution Networks].

To fan out to large, geographically dispersed audiences of 10,000 or more (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get content cached in locations that are geographically close to your listeners. By far the easiest (read: most cost effective) way to do this at scale is to leverage the already existing infrastructure of CDNs (companies like Akamai or CloudFlare, that have a world-wide footprint). That means using streaming formats that utilize segmented distribution mechanisms, such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast by using relays, but it?s complex to configure and monitor while being not well supported at many CDNs (Akamai for example discontinued their IceCast product offering several years ago). HLS OTOH plays very well with that infrastructure because it?s effectively just a bunch of static files that get replicated via HTTP[S]. No special ?server? software is required; bog-standard Apache or Nginx work just fine, because the complex ?media handling? bits have been intentionally pushed to the endpoints; namely the encoder and (especially) the players. Today though, when FOSS HLS audio encoders are available and pretty much every browser supports playing HLS content natively, the complexity angle can be largely ignored by content creators.

Just my take. That, and 2 ? will get you a (cheap) cup of coffee?

Cheers!


|---------------------------------------------------------------------|
| Frederick F. Gleason, Jr. |             Chief Developer             |
|                           |             Paravel Systems             |
|---------------------------------------------------------------------|
|    All progress is based upon a universal innate desire of every    |
|    organism to live beyond its income.                              |
|                                                                     |
|                                                  -- Samuel Butler   |
|                                                  "Notebooks"        |
|---------------------------------------------------------------------|

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240320/94638559/attachment.htm>

From thomas.zumbrunnen at gmail.com  Wed Mar 20 20:50:31 2024
From: thomas.zumbrunnen at gmail.com (thomas.zumbrunnen at gmail.com)
Date: Wed, 20 Mar 2024 21:50:31 +0100
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
Message-ID: <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>

Dear all

 

My 5 cents (or Rappen in CH) if it comes to serving many clients.

We are running a 4 node cluster since several years ? rock solid and w/o any issues. This cluster serves many thousands of listeners from all over the world. Our source transcoder sending the audio streams to each Node.  Hence, transcoding power is not an issue here. The four Nodes a geographically dispersed in 3 countries in Europe. In our case each Node is running Debian with icecast and has 10Gbit connectivity with brilliant worldwide peerings. Good peering is key, choose your ISP wisely ?.Each icecast servers has the same multi domain ssl cert. which allows us to deliver to several customers (each customer a subdomain) the cluster is round robin load balanced by using AWS Route53. This approach may can be achived also with other DNS Providers like Cloudflare. For example, if one node need to be taken down for maintenance, Route53 throws the Node out of the DNS automatically. This will be achived with ?health checks? This mechanism is pretty fast and responsive. If a client gets disconnected and tries a reconnect, the RR DNS is passing the client immediately to a working Node. No issues here as well. It just works.

 

In the beginning we?ve experienced similar issues even though, the bandwidth capacity of the VM?s was never the root cause. We?ve identified some solutions:

 

1.	Linux TCP stack tuning. Cloudflare has many studies published in their Blogs about this. But you will find a lot about this tuning also in the interweb
2.	Consider to bake your own kernel, which is tuned for high throughput ? goes in line with TCP Stack tunning
3.	Tune the Linux open file limits and adjust the init start script for the icecast server. Example : start icecast with  ulimit -c unlimited and ulimit -n 32768
4.	Consider to use FreeBSD instead of Linux. FreeBSD has the better TCP stack out of the box.
5.	If all of this is not feasible for you, just add a new Node to the cluster and level the amount of clients to more Nodes.

 

For the points 1. and 2. I can?t give you a ?out of the box? solution or default settings. It?s an iteration process: adjusting, trial, load testing, monitoring and <repeat>

Because the result will need to fit your requirements, therefore every setup might need a different tuning. And btw. do not try using icecast on Windows Servers, if you need to serve a lot of clients ? 

 

Happy icecasting

tom

 

Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Fred Gleason
Gesendet: Mittwoch, 20. M?rz 2024 20:53
An: Icecast streaming server user discussions <icecast at xiph.org>
Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)

 

On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com <mailto:wayne at cffcs.com> > wrote:

 

In Windows and Linux web servers, we can create a forest for our web servers.
Send traffic to different servers to even the workload.

Can we do something like this with the Icecast servers?
(or)
Will we have to install new VMs, add the heavy stations on that one,
and send the new traffic there?

 

Ok, I?m going to be ?that guy??

 

I would argue that, as soon as you?ve hit an audience size of 10,000 or more (especially if that audience is at all geographically dispersed), IceCast is basically off the table. The reason why can be summarized in three letters: ?CDN? [Content Distribution Networks].

 

To fan out to large, geographically dispersed audiences of 10,000 or more (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get content cached in locations that are geographically close to your listeners. By far the easiest (read: most cost effective) way to do this at scale is to leverage the already existing infrastructure of CDNs (companies like Akamai or CloudFlare, that have a world-wide footprint). That means using streaming formats that utilize segmented distribution mechanisms, such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast by using relays, but it?s complex to configure and monitor while being not well supported at many CDNs (Akamai for example discontinued their IceCast product offering several years ago). HLS OTOH plays very well with that infrastructure because it?s effectively just a bunch of static files that get replicated via HTTP[S]. No special ?server? software is required; bog-standard Apache or Nginx work just fine, because the complex ?media handling? bits have been intentionally pushed to the endpoints; namely the encoder and (especially) the players. Today though, when FOSS HLS audio encoders are available and pretty much every browser supports playing HLS content natively, the complexity angle can be largely ignored by content creators.

 

Just my take. That, and 2 ? will get you a (cheap) cup of coffee?

 

Cheers!

 

 

|---------------------------------------------------------------------|

| Frederick F. Gleason, Jr. |             Chief Developer             |

|                           |             Paravel Systems             |

|---------------------------------------------------------------------|

|    All progress is based upon a universal innate desire of every    |

|    organism to live beyond its income.                              |

|                                                                     |

|                                                  -- Samuel Butler   |

|                                                  "Notebooks"        |

|---------------------------------------------------------------------|

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240320/d626b96f/attachment.htm>

From wayne at cffcs.com  Wed Mar 20 22:05:58 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Wed, 20 Mar 2024 18:05:58 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
Message-ID: <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>

Tom and Frederick.
Thank you both for your input. It is greatly appreciated, not only by me,
but I am sure for many others who find this thread.

Tom - using Linux server for icecast.

HLS was brought to my attention a while back on the liquidsoap forum.
I have not had a chance to completely look in on it, but do plan on
checking it out.

Tom. You have your icecast servers in a round robin configuration.
Are you using NGINX for that?
I have watched several YouTube videos on using it for doing round robin as
well as ssl and other things.

I understand that once a scenario of say 10,000 or more, that I will have
to look into either a bigger infrastructure on my side or leasing space on
the cloud somewhere.

Now, if I did do the cloud.
Would that be to host the servers or just the files or everything?

Wayne


On Wed, Mar 20, 2024, 4:50?PM <thomas.zumbrunnen at gmail.com> wrote:

> Dear all
>
>
>
> My 5 cents (or Rappen in CH) if it comes to serving many clients.
>
> We are running a 4 node cluster since several years ? rock solid and w/o
> any issues. This cluster serves many thousands of listeners from all over
> the world. Our source transcoder sending the audio streams to each Node.
> Hence, transcoding power is not an issue here. The four Nodes a
> geographically dispersed in 3 countries in Europe. In our case each Node
> is running Debian with icecast and has 10Gbit connectivity with brilliant
> worldwide peerings. Good peering is key, choose your ISP wisely ?.Each
> icecast servers has the same multi domain ssl cert. which allows us to
> deliver to several customers (each customer a subdomain) the cluster is
> round robin load balanced by using AWS Route53. This approach may can be
> achived also with other DNS Providers like Cloudflare. For example, if one
> node need to be taken down for maintenance, Route53 throws the Node out of
> the DNS automatically. This will be achived with ?health checks? This
> mechanism is pretty fast and responsive. If a client gets disconnected and
> tries a reconnect, the RR DNS is passing the client immediately to a
> working Node. No issues here as well. It just works.
>
>
>
> In the beginning we?ve experienced similar issues even though, the
> bandwidth capacity of the VM?s was never the root cause. We?ve identified
> some solutions:
>
>
>
>    1. Linux TCP stack tuning. Cloudflare has many studies published in
>    their Blogs about this. But you will find a lot about this tuning also in
>    the interweb
>    2. Consider to bake your own kernel, which is tuned for high
>    throughput ? goes in line with TCP Stack tunning
>    3. Tune the Linux open file limits and adjust the init start script
>    for the icecast server. Example : start icecast with  ulimit -c unlimited
>    and ulimit -n 32768
>    4. Consider to use FreeBSD instead of Linux. FreeBSD has the better
>    TCP stack out of the box.
>    5. If all of this is not feasible for you, just add a new Node to the
>    cluster and level the amount of clients to more Nodes.
>
>
>
> For the points 1. and 2. I can?t give you a ?out of the box? solution or
> default settings. It?s an iteration process: adjusting, trial, load
> testing, monitoring and <repeat>
>
> Because the result will need to fit your requirements, therefore every
> setup might need a different tuning. And btw. do not try using icecast on
> Windows Servers, if you need to serve a lot of clients ?
>
>
>
> Happy icecasting
>
> tom
>
>
>
> *Von:* Icecast <icecast-bounces at xiph.org> *Im Auftrag von *Fred Gleason
> *Gesendet:* Mittwoch, 20. M?rz 2024 20:53
> *An:* Icecast streaming server user discussions <icecast at xiph.org>
> *Betreff:* Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of
> listeners. (What kind of infrastructure)
>
>
>
> On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com> wrote:
>
>
>
> In Windows and Linux web servers, we can create a forest for our web
> servers.
> Send traffic to different servers to even the workload.
>
> Can we do something like this with the Icecast servers?
> (or)
> Will we have to install new VMs, add the heavy stations on that one,
> and send the new traffic there?
>
>
>
> Ok, I?m going to be ?that guy??
>
>
>
> I would argue that, as soon as you?ve hit an audience size of 10,000 or
> more (especially if that audience is at all geographically dispersed),
> IceCast is basically off the table. The reason why can be summarized in
> three letters: ?CDN? [Content Distribution Networks].
>
>
>
> To fan out to large, geographically dispersed audiences of 10,000 or more
> (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get
> content cached in locations that are geographically close to your
> listeners. By far the easiest (read: most cost effective) way to do this at
> scale is to leverage the already existing infrastructure of CDNs (companies
> like Akamai or CloudFlare, that have a world-wide footprint). That means
> using streaming formats that utilize segmented distribution mechanisms,
> such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast
> by using relays, but it?s complex to configure and monitor while being not
> well supported at many CDNs (Akamai for example discontinued their IceCast
> product offering several years ago). HLS OTOH plays very well with that
> infrastructure because it?s effectively just a bunch of static files that
> get replicated via HTTP[S]. No special ?server? software is required;
> bog-standard Apache or Nginx work just fine, because the complex ?media
> handling? bits have been intentionally pushed to the endpoints; namely the
> encoder and (especially) the players. Today though, when FOSS HLS audio
> encoders are available and pretty much every browser supports playing HLS
> content natively, the complexity angle can be largely ignored by content
> creators.
>
>
>
> Just my take. That, and 2 ? will get you a (cheap) cup of coffee?
>
>
>
> Cheers!
>
>
>
>
>
> |---------------------------------------------------------------------|
>
> | Frederick F. Gleason, Jr. |             Chief Developer             |
>
> |                           |             Paravel Systems             |
>
> |---------------------------------------------------------------------|
>
> |    All progress is based upon a universal innate desire of every    |
>
> |    organism to live beyond its income.                              |
>
> |                                                                     |
>
> |                                                  -- Samuel Butler   |
>
> |                                                  "Notebooks"        |
>
> |---------------------------------------------------------------------|
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240320/187fc23b/attachment.htm>

From fredg at paravelsystems.com  Thu Mar 21 00:49:48 2024
From: fredg at paravelsystems.com (Fred Gleason)
Date: Wed, 20 Mar 2024 20:49:48 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
Message-ID: <665505BD-946B-4837-9514-627C340A9382@paravelsystems.com>

On Mar 20, 2024, at 18:05, Wayne Barron <wayne at cffcs.com> wrote:

> Now, if I did do the cloud.
> Would that be to host the servers or just the files or everything?

A lot depends on the composition of your audience. If you?re looking to serve a primarily regional audience (as many terrestrial AM/FM outlets do in the US), then Icecast paired with an ISP that has good coverage of the target region can be a very effective setup. I do echo Tom?s advice about good peering arrangements!

You want to investigate costs carefully when looking at cloud setups ? I?ve found that bandwidth (as opposed to raw IOPS) at some cloud providers can be quite expensive. One of the nice things about using a CDN is that you only need to deliver the stream to a single (or perhaps redundant pair of) endpoint(s). You can get by with quite a modest circuit for the ?first mile? that way, though of course you will be paying for the bandwidth provided by the CDN on the fanout.

Cheers!


|---------------------------------------------------------------------|
| Frederick F. Gleason, Jr. |             Chief Developer             |
|                           |             Paravel Systems             |
|---------------------------------------------------------------------|
|         A room without books is like a body without a soul.         |
|                                                                     |
|                                                         -- Cicero   |
|---------------------------------------------------------------------|
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240320/2d11dad5/attachment.htm>

From thomas.zumbrunnen at gmail.com  Thu Mar 21 08:08:31 2024
From: thomas.zumbrunnen at gmail.com (thomas.zumbrunnen at gmail.com)
Date: Thu, 21 Mar 2024 09:08:31 +0100
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
Message-ID: <00ab01da7b66$f7712900$e6537b00$@gmail.com>

Hi Wayne

 

Yep, HLS is something we would like to use with Icecast. As you mentioned, Liquidsoap made a great step in the right direction if it comes to HLS. It seem that icecast might not get this support soon, as you might read in the discussion list. I don?t share these arguments, because we have real world use cases (we have many of them) where we  would like use icecast with HLS support. But unfortunately we needed to go with another streaming server application which supports HLS to fulfill our requirements.

 

But coming back to the topic of this thread.

I was looking into using NGINX as a reverse proxy for doing the load balancing (single point of SSL and centralized monitoring). NGINX is awesome in doing this. And there are some valid arguments using the approach to put a single NGINX server (or a two node Nginx Cluster) in front of a multi node icecast cluster. At that time, 5 years ago,  we decided to go with the ? poor man solution ? and use the RR DNS approach. We did not had enough time for investigation, development and testing. Additional, we eliminated another single point of failure. Sometimes I like way of KISS (keep it simple stupid) ?

 

if you need to serve static files globaly, I would consider the CDN approach. We use CF for this, but any other CDN provider would do the job. They are kind of ?same same - but different?

Cheers

Tom

 

Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Wayne Barron
Gesendet: Mittwoch, 20. M?rz 2024 23:06
An: Icecast streaming server user discussions <icecast at xiph.org>
Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)

 

Tom and Frederick.

Thank you both for your input. It is greatly appreciated, not only by me, but I am sure for many others who find this thread.

 

Tom - using Linux server for icecast.

 

HLS was brought to my attention a while back on the liquidsoap forum.

I have not had a chance to completely look in on it, but do plan on checking it out.

 

Tom. You have your icecast servers in a round robin configuration.

Are you using NGINX for that?

I have watched several YouTube videos on using it for doing round robin as well as ssl and other things.

 

I understand that once a scenario of say 10,000 or more, that I will have to look into either a bigger infrastructure on my side or leasing space on the cloud somewhere.

 

Now, if I did do the cloud.

Would that be to host the servers or just the files or everything?

 

Wayne 

 

On Wed, Mar 20, 2024, 4:50?PM <thomas.zumbrunnen at gmail.com <mailto:thomas.zumbrunnen at gmail.com> > wrote:

Dear all

 

My 5 cents (or Rappen in CH) if it comes to serving many clients.

We are running a 4 node cluster since several years ? rock solid and w/o any issues. This cluster serves many thousands of listeners from all over the world. Our source transcoder sending the audio streams to each Node.  Hence, transcoding power is not an issue here. The four Nodes a geographically dispersed in 3 countries in Europe. In our case each Node is running Debian with icecast and has 10Gbit connectivity with brilliant worldwide peerings. Good peering is key, choose your ISP wisely ?.Each icecast servers has the same multi domain ssl cert. which allows us to deliver to several customers (each customer a subdomain) the cluster is round robin load balanced by using AWS Route53. This approach may can be achived also with other DNS Providers like Cloudflare. For example, if one node need to be taken down for maintenance, Route53 throws the Node out of the DNS automatically. This will be achived with ?health checks? This mechanism is pretty fast and responsive. If a client gets disconnected and tries a reconnect, the RR DNS is passing the client immediately to a working Node. No issues here as well. It just works.

 

In the beginning we?ve experienced similar issues even though, the bandwidth capacity of the VM?s was never the root cause. We?ve identified some solutions:

 

1.	Linux TCP stack tuning. Cloudflare has many studies published in their Blogs about this. But you will find a lot about this tuning also in the interweb
2.	Consider to bake your own kernel, which is tuned for high throughput ? goes in line with TCP Stack tunning
3.	Tune the Linux open file limits and adjust the init start script for the icecast server. Example : start icecast with  ulimit -c unlimited and ulimit -n 32768
4.	Consider to use FreeBSD instead of Linux. FreeBSD has the better TCP stack out of the box.
5.	If all of this is not feasible for you, just add a new Node to the cluster and level the amount of clients to more Nodes.

 

For the points 1. and 2. I can?t give you a ?out of the box? solution or default settings. It?s an iteration process: adjusting, trial, load testing, monitoring and <repeat>

Because the result will need to fit your requirements, therefore every setup might need a different tuning. And btw. do not try using icecast on Windows Servers, if you need to serve a lot of clients ? 

 

Happy icecasting

tom

 

Von: Icecast <icecast-bounces at xiph.org <mailto:icecast-bounces at xiph.org> > Im Auftrag von Fred Gleason
Gesendet: Mittwoch, 20. M?rz 2024 20:53
An: Icecast streaming server user discussions <icecast at xiph.org <mailto:icecast at xiph.org> >
Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)

 

On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com <mailto:wayne at cffcs.com> > wrote:

 

In Windows and Linux web servers, we can create a forest for our web servers.
Send traffic to different servers to even the workload.

Can we do something like this with the Icecast servers?
(or)
Will we have to install new VMs, add the heavy stations on that one,
and send the new traffic there?

 

Ok, I?m going to be ?that guy??

 

I would argue that, as soon as you?ve hit an audience size of 10,000 or more (especially if that audience is at all geographically dispersed), IceCast is basically off the table. The reason why can be summarized in three letters: ?CDN? [Content Distribution Networks].

 

To fan out to large, geographically dispersed audiences of 10,000 or more (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get content cached in locations that are geographically close to your listeners. By far the easiest (read: most cost effective) way to do this at scale is to leverage the already existing infrastructure of CDNs (companies like Akamai or CloudFlare, that have a world-wide footprint). That means using streaming formats that utilize segmented distribution mechanisms, such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast by using relays, but it?s complex to configure and monitor while being not well supported at many CDNs (Akamai for example discontinued their IceCast product offering several years ago). HLS OTOH plays very well with that infrastructure because it?s effectively just a bunch of static files that get replicated via HTTP[S]. No special ?server? software is required; bog-standard Apache or Nginx work just fine, because the complex ?media handling? bits have been intentionally pushed to the endpoints; namely the encoder and (especially) the players. Today though, when FOSS HLS audio encoders are available and pretty much every browser supports playing HLS content natively, the complexity angle can be largely ignored by content creators.

 

Just my take. That, and 2 ? will get you a (cheap) cup of coffee?

 

Cheers!

 

 

|---------------------------------------------------------------------|

| Frederick F. Gleason, Jr. |             Chief Developer             |

|                           |             Paravel Systems             |

|---------------------------------------------------------------------|

|    All progress is based upon a universal innate desire of every    |

|    organism to live beyond its income.                              |

|                                                                     |

|                                                  -- Samuel Butler   |

|                                                  "Notebooks"        |

|---------------------------------------------------------------------|

 

_______________________________________________
Icecast mailing list
Icecast at xiph.org <mailto:Icecast at xiph.org> 
http://lists.xiph.org/mailman/listinfo/icecast


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240321/636ef45e/attachment.htm>

From wayne at cffcs.com  Thu Mar 21 10:48:16 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Thu, 21 Mar 2024 06:48:16 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <00ab01da7b66$f7712900$e6537b00$@gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
Message-ID: <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>

Looking through HLS, I found this.
https://github.com/mbugeia/srt2hls
For what I can un
As in the liquidsoap link I shared, this is what my links look like.
https://github.com/savonet/liquidsoap/discussions/3024
Liquisoap link.
music = playlist("https://www.domain.com/Panels/DJ/101.asp",mode="normal")

On Thu, Mar 21, 2024 at 4:08?AM <thomas.zumbrunnen at gmail.com> wrote:
>
> Hi Wayne
>
>
>
> Yep, HLS is something we would like to use with Icecast. As you mentioned, Liquidsoap made a great step in the right direction if it comes to HLS. It seem that icecast might not get this support soon, as you might read in the discussion list. I don?t share these arguments, because we have real world use cases (we have many of them) where we  would like use icecast with HLS support. But unfortunately we needed to go with another streaming server application which supports HLS to fulfill our requirements.
>
>
>
> But coming back to the topic of this thread.
>
> I was looking into using NGINX as a reverse proxy for doing the load balancing (single point of SSL and centralized monitoring). NGINX is awesome in doing this. And there are some valid arguments using the approach to put a single NGINX server (or a two node Nginx Cluster) in front of a multi node icecast cluster. At that time, 5 years ago,  we decided to go with the ? poor man solution ? and use the RR DNS approach. We did not had enough time for investigation, development and testing. Additional, we eliminated another single point of failure. Sometimes I like way of KISS (keep it simple stupid) ?
>
>
>
> if you need to serve static files globaly, I would consider the CDN approach. We use CF for this, but any other CDN provider would do the job. They are kind of ?same same - but different?
>
> Cheers
>
> Tom
>
>
>
> Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Wayne Barron
> Gesendet: Mittwoch, 20. M?rz 2024 23:06
> An: Icecast streaming server user discussions <icecast at xiph.org>
> Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)
>
>
>
> Tom and Frederick.
>
> Thank you both for your input. It is greatly appreciated, not only by me, but I am sure for many others who find this thread.
>
>
>
> Tom - using Linux server for icecast.
>
>
>
> HLS was brought to my attention a while back on the liquidsoap forum.
>
> I have not had a chance to completely look in on it, but do plan on checking it out.
>
>
>
> Tom. You have your icecast servers in a round robin configuration.
>
> Are you using NGINX for that?
>
> I have watched several YouTube videos on using it for doing round robin as well as ssl and other things.
>
>
>
> I understand that once a scenario of say 10,000 or more, that I will have to look into either a bigger infrastructure on my side or leasing space on the cloud somewhere.
>
>
>
> Now, if I did do the cloud.
>
> Would that be to host the servers or just the files or everything?
>
>
>
> Wayne
>
>
>
> On Wed, Mar 20, 2024, 4:50?PM <thomas.zumbrunnen at gmail.com> wrote:
>
> Dear all
>
>
>
> My 5 cents (or Rappen in CH) if it comes to serving many clients.
>
> We are running a 4 node cluster since several years ? rock solid and w/o any issues. This cluster serves many thousands of listeners from all over the world. Our source transcoder sending the audio streams to each Node.  Hence, transcoding power is not an issue here. The four Nodes a geographically dispersed in 3 countries in Europe. In our case each Node is running Debian with icecast and has 10Gbit connectivity with brilliant worldwide peerings. Good peering is key, choose your ISP wisely ?.Each icecast servers has the same multi domain ssl cert. which allows us to deliver to several customers (each customer a subdomain) the cluster is round robin load balanced by using AWS Route53. This approach may can be achived also with other DNS Providers like Cloudflare. For example, if one node need to be taken down for maintenance, Route53 throws the Node out of the DNS automatically. This will be achived with ?health checks? This mechanism is pretty fast and responsive. If a client gets disconnected and tries a reconnect, the RR DNS is passing the client immediately to a working Node. No issues here as well. It just works.
>
>
>
> In the beginning we?ve experienced similar issues even though, the bandwidth capacity of the VM?s was never the root cause. We?ve identified some solutions:
>
>
>
> Linux TCP stack tuning. Cloudflare has many studies published in their Blogs about this. But you will find a lot about this tuning also in the interweb
> Consider to bake your own kernel, which is tuned for high throughput ? goes in line with TCP Stack tunning
> Tune the Linux open file limits and adjust the init start script for the icecast server. Example : start icecast with  ulimit -c unlimited and ulimit -n 32768
> Consider to use FreeBSD instead of Linux. FreeBSD has the better TCP stack out of the box.
> If all of this is not feasible for you, just add a new Node to the cluster and level the amount of clients to more Nodes.
>
>
>
> For the points 1. and 2. I can?t give you a ?out of the box? solution or default settings. It?s an iteration process: adjusting, trial, load testing, monitoring and <repeat>
>
> Because the result will need to fit your requirements, therefore every setup might need a different tuning. And btw. do not try using icecast on Windows Servers, if you need to serve a lot of clients ?
>
>
>
> Happy icecasting
>
> tom
>
>
>
> Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Fred Gleason
> Gesendet: Mittwoch, 20. M?rz 2024 20:53
> An: Icecast streaming server user discussions <icecast at xiph.org>
> Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)
>
>
>
> On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com> wrote:
>
>
>
> In Windows and Linux web servers, we can create a forest for our web servers.
> Send traffic to different servers to even the workload.
>
> Can we do something like this with the Icecast servers?
> (or)
> Will we have to install new VMs, add the heavy stations on that one,
> and send the new traffic there?
>
>
>
> Ok, I?m going to be ?that guy??
>
>
>
> I would argue that, as soon as you?ve hit an audience size of 10,000 or more (especially if that audience is at all geographically dispersed), IceCast is basically off the table. The reason why can be summarized in three letters: ?CDN? [Content Distribution Networks].
>
>
>
> To fan out to large, geographically dispersed audiences of 10,000 or more (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get content cached in locations that are geographically close to your listeners. By far the easiest (read: most cost effective) way to do this at scale is to leverage the already existing infrastructure of CDNs (companies like Akamai or CloudFlare, that have a world-wide footprint). That means using streaming formats that utilize segmented distribution mechanisms, such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast by using relays, but it?s complex to configure and monitor while being not well supported at many CDNs (Akamai for example discontinued their IceCast product offering several years ago). HLS OTOH plays very well with that infrastructure because it?s effectively just a bunch of static files that get replicated via HTTP[S]. No special ?server? software is required; bog-standard Apache or Nginx work just fine, because the complex ?media handling? bits have been intentionally pushed to the endpoints; namely the encoder and (especially) the players. Today though, when FOSS HLS audio encoders are available and pretty much every browser supports playing HLS content natively, the complexity angle can be largely ignored by content creators.
>
>
>
> Just my take. That, and 2 ? will get you a (cheap) cup of coffee?
>
>
>
> Cheers!
>
>
>
>
>
> |---------------------------------------------------------------------|
>
> | Frederick F. Gleason, Jr. |             Chief Developer             |
>
> |                           |             Paravel Systems             |
>
> |---------------------------------------------------------------------|
>
> |    All progress is based upon a universal innate desire of every    |
>
> |    organism to live beyond its income.                              |
>
> |                                                                     |
>
> |                                                  -- Samuel Butler   |
>
> |                                                  "Notebooks"        |
>
> |---------------------------------------------------------------------|
>
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast


From hick.icecast at gink.org  Thu Mar 21 13:02:49 2024
From: hick.icecast at gink.org (gARetH baBB)
Date: Thu, 21 Mar 2024 13:02:49 +0000 (GMT)
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
 <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>
Message-ID: <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>

On Thu, 21 Mar 2024, Wayne Barron wrote:

> Looking through HLS, I found this.
> https://github.com/mbugeia/srt2hls

Or you could just use ffmpeg:

ffmpeg -hide_banner -i "${icecaststream}" -c:a copy -vn -strftime 1
-f hls -hls_time 6 -hls_list_size 10 -hls_segment_filename "${hlspath}radio-%Y%m%d-%s.ts" 
-hls_flags delete_segments -segment_format mpegts "${hlspath}radio.m3u8"

then you just have a set of files for a normal webserver.

Note the c:a copy means there is no transcoding in this instance.

If theory the HLS spec only allows for mp3 or aac, but I've abused it by 
even using opus and not had a problem with the limited number of clients 
I've tried.

ffmpeg also does DASH.


From wayne at cffcs.com  Thu Mar 21 16:41:41 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Thu, 21 Mar 2024 12:41:41 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <00ab01da7b66$f7712900$e6537b00$@gmail.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
Message-ID: <CAJatk5mcG=9yKToPp3AsaTbK8qp7sgESic_ZBGxMo-AUHq_Oyg@mail.gmail.com>

>From what I understand, I should be able to take a Dynamic Source.
Which will be from an ASP file with a list of songs.
Example of the list within the ASP File.

https://domain/Asylum/01082021184726649.mp3
https://domain/Asylum/01082021184727383.mp3
https://domain/Asylum/01082021184727946.mp3

And HLS will create a live one.m3u8 files in the /tmp/HLS directory.
Every time I update the ASP page, it updates the live.m3u8 file in the
HLS directory.

I found this page. (But it is NOT active)
https://github.com/mbugeia/srt2hls
Toots has implemented HLS Metadata in Liquidsoap for rolling-release-v2.2.x!
https://github.com/savonet/liquidsoap/issues/1487#issuecomment-1605797838

It took a while to get used to working with Icecast and Liquidsoap,
and now I will have to learn HLS and Liquidsoap and how to make them
work together.

It would be great if anyone knew of a page that gives instructions on
how to set up a radio station with HLS and Liquidsoap that can at
least get me started.

This is pretty much the way my liq file is set up with Liquidsoap and Icecast.

# !/home/iceadmin/.opam/default/bin/liquidsoap
# log dir
set ("log.file.path","/var/log/liquidsoap/Radio.log")
set("scheduler.fast_queues",1)
# Music
music = playlist("https://www.example.com/Panels/DJ/101.asp",mode="normal")

# Start building the feed with music
radio = music
# Now add some jingles
#radio = random(weights = [1, 4],[radio, jingles])

# Stream it out
output.icecast(%mp3(bitrate=256,samplerate=44100,internal_quality=0,id3v2=true,stereo=true,stereo_mode="stereo"),
name="Radio-101",
encoding="UTF-8",
host="192.168.2.203",
port=8000,
password="hackme",
icy_metadata="true",
description="Where I come to Rock!",
mount="Radio-101",
mksafe(radio))

Can I still use this, or will I have to add something to it to make it
work with HLS?
I see that there is one area that would need to be changed.
output.icecast
And this makes me think that the entire script will not work with HLS,
or will it?

A lot to learn and a lot to think about.
I just launched for Linux VM for the first time in over a year, and I
am trying to remember where everything is at and how to work it again
with Icecast and Liquidsoap.
Luckily, I have a lot of the things written down for reference.

On Thu, Mar 21, 2024 at 4:08?AM <thomas.zumbrunnen at gmail.com> wrote:
>
> Hi Wayne
>
>
>
> Yep, HLS is something we would like to use with Icecast. As you mentioned, Liquidsoap made a great step in the right direction if it comes to HLS. It seem that icecast might not get this support soon, as you might read in the discussion list. I don?t share these arguments, because we have real world use cases (we have many of them) where we  would like use icecast with HLS support. But unfortunately we needed to go with another streaming server application which supports HLS to fulfill our requirements.
>
>
>
> But coming back to the topic of this thread.
>
> I was looking into using NGINX as a reverse proxy for doing the load balancing (single point of SSL and centralized monitoring). NGINX is awesome in doing this. And there are some valid arguments using the approach to put a single NGINX server (or a two node Nginx Cluster) in front of a multi node icecast cluster. At that time, 5 years ago,  we decided to go with the ? poor man solution ? and use the RR DNS approach. We did not had enough time for investigation, development and testing. Additional, we eliminated another single point of failure. Sometimes I like way of KISS (keep it simple stupid) ?
>
>
>
> if you need to serve static files globaly, I would consider the CDN approach. We use CF for this, but any other CDN provider would do the job. They are kind of ?same same - but different?
>
> Cheers
>
> Tom
>
>
>
> Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Wayne Barron
> Gesendet: Mittwoch, 20. M?rz 2024 23:06
> An: Icecast streaming server user discussions <icecast at xiph.org>
> Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)
>
>
>
> Tom and Frederick.
>
> Thank you both for your input. It is greatly appreciated, not only by me, but I am sure for many others who find this thread.
>
>
>
> Tom - using Linux server for icecast.
>
>
>
> HLS was brought to my attention a while back on the liquidsoap forum.
>
> I have not had a chance to completely look in on it, but do plan on checking it out.
>
>
>
> Tom. You have your icecast servers in a round robin configuration.
>
> Are you using NGINX for that?
>
> I have watched several YouTube videos on using it for doing round robin as well as ssl and other things.
>
>
>
> I understand that once a scenario of say 10,000 or more, that I will have to look into either a bigger infrastructure on my side or leasing space on the cloud somewhere.
>
>
>
> Now, if I did do the cloud.
>
> Would that be to host the servers or just the files or everything?
>
>
>
> Wayne
>
>
>
> On Wed, Mar 20, 2024, 4:50?PM <thomas.zumbrunnen at gmail.com> wrote:
>
> Dear all
>
>
>
> My 5 cents (or Rappen in CH) if it comes to serving many clients.
>
> We are running a 4 node cluster since several years ? rock solid and w/o any issues. This cluster serves many thousands of listeners from all over the world. Our source transcoder sending the audio streams to each Node.  Hence, transcoding power is not an issue here. The four Nodes a geographically dispersed in 3 countries in Europe. In our case each Node is running Debian with icecast and has 10Gbit connectivity with brilliant worldwide peerings. Good peering is key, choose your ISP wisely ?.Each icecast servers has the same multi domain ssl cert. which allows us to deliver to several customers (each customer a subdomain) the cluster is round robin load balanced by using AWS Route53. This approach may can be achived also with other DNS Providers like Cloudflare. For example, if one node need to be taken down for maintenance, Route53 throws the Node out of the DNS automatically. This will be achived with ?health checks? This mechanism is pretty fast and responsive. If a client gets disconnected and tries a reconnect, the RR DNS is passing the client immediately to a working Node. No issues here as well. It just works.
>
>
>
> In the beginning we?ve experienced similar issues even though, the bandwidth capacity of the VM?s was never the root cause. We?ve identified some solutions:
>
>
>
> Linux TCP stack tuning. Cloudflare has many studies published in their Blogs about this. But you will find a lot about this tuning also in the interweb
> Consider to bake your own kernel, which is tuned for high throughput ? goes in line with TCP Stack tunning
> Tune the Linux open file limits and adjust the init start script for the icecast server. Example : start icecast with  ulimit -c unlimited and ulimit -n 32768
> Consider to use FreeBSD instead of Linux. FreeBSD has the better TCP stack out of the box.
> If all of this is not feasible for you, just add a new Node to the cluster and level the amount of clients to more Nodes.
>
>
>
> For the points 1. and 2. I can?t give you a ?out of the box? solution or default settings. It?s an iteration process: adjusting, trial, load testing, monitoring and <repeat>
>
> Because the result will need to fit your requirements, therefore every setup might need a different tuning. And btw. do not try using icecast on Windows Servers, if you need to serve a lot of clients ?
>
>
>
> Happy icecasting
>
> tom
>
>
>
> Von: Icecast <icecast-bounces at xiph.org> Im Auftrag von Fred Gleason
> Gesendet: Mittwoch, 20. M?rz 2024 20:53
> An: Icecast streaming server user discussions <icecast at xiph.org>
> Betreff: Re: [Icecast] Education - 1, 000s, 100, 000's, Millions of listeners. (What kind of infrastructure)
>
>
>
> On Mar 20, 2024, at 13:16, Wayne Barron <wayne at cffcs.com> wrote:
>
>
>
> In Windows and Linux web servers, we can create a forest for our web servers.
> Send traffic to different servers to even the workload.
>
> Can we do something like this with the Icecast servers?
> (or)
> Will we have to install new VMs, add the heavy stations on that one,
> and send the new traffic there?
>
>
>
> Ok, I?m going to be ?that guy??
>
>
>
> I would argue that, as soon as you?ve hit an audience size of 10,000 or more (especially if that audience is at all geographically dispersed), IceCast is basically off the table. The reason why can be summarized in three letters: ?CDN? [Content Distribution Networks].
>
>
>
> To fan out to large, geographically dispersed audiences of 10,000 or more (not to mention 100k?s or, Lord help us, 1M?s or more), you need to get content cached in locations that are geographically close to your listeners. By far the easiest (read: most cost effective) way to do this at scale is to leverage the already existing infrastructure of CDNs (companies like Akamai or CloudFlare, that have a world-wide footprint). That means using streaming formats that utilize segmented distribution mechanisms, such as HLS or DASH. You can kinda-sorta do this sort of thing with IceCast by using relays, but it?s complex to configure and monitor while being not well supported at many CDNs (Akamai for example discontinued their IceCast product offering several years ago). HLS OTOH plays very well with that infrastructure because it?s effectively just a bunch of static files that get replicated via HTTP[S]. No special ?server? software is required; bog-standard Apache or Nginx work just fine, because the complex ?media handling? bits have been intentionally pushed to the endpoints; namely the encoder and (especially) the players. Today though, when FOSS HLS audio encoders are available and pretty much every browser supports playing HLS content natively, the complexity angle can be largely ignored by content creators.
>
>
>
> Just my take. That, and 2 ? will get you a (cheap) cup of coffee?
>
>
>
> Cheers!
>
>
>
>
>
> |---------------------------------------------------------------------|
>
> | Frederick F. Gleason, Jr. |             Chief Developer             |
>
> |                           |             Paravel Systems             |
>
> |---------------------------------------------------------------------|
>
> |    All progress is based upon a universal innate desire of every    |
>
> |    organism to live beyond its income.                              |
>
> |                                                                     |
>
> |                                                  -- Samuel Butler   |
>
> |                                                  "Notebooks"        |
>
> |---------------------------------------------------------------------|
>
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast


From wayne at cffcs.com  Thu Mar 21 16:44:10 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Thu, 21 Mar 2024 12:44:10 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
 <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>
 <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>
Message-ID: <CAJatk5kpkWZdVxdYcWW2vRv15kmsLHq_dJ2ji-rAfgTv_UR6Ng@mail.gmail.com>

gARetH
My plans are to have live DJs in the future.
Will ffmpeg work with live DJs?
I know the Liquidsoap does.

On Thu, Mar 21, 2024 at 9:24?AM gARetH baBB <hick.icecast at gink.org> wrote:
>
> On Thu, 21 Mar 2024, Wayne Barron wrote:
>
> > Looking through HLS, I found this.
> > https://github.com/mbugeia/srt2hls
>
> Or you could just use ffmpeg:
>
> ffmpeg -hide_banner -i "${icecaststream}" -c:a copy -vn -strftime 1
> -f hls -hls_time 6 -hls_list_size 10 -hls_segment_filename "${hlspath}radio-%Y%m%d-%s.ts"
> -hls_flags delete_segments -segment_format mpegts "${hlspath}radio.m3u8"
>
> then you just have a set of files for a normal webserver.
>
> Note the c:a copy means there is no transcoding in this instance.
>
> If theory the HLS spec only allows for mp3 or aac, but I've abused it by
> even using opus and not had a problem with the limited number of clients
> I've tried.
>
> ffmpeg also does DASH.
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast


From fredg at paravelsystems.com  Thu Mar 21 16:53:02 2024
From: fredg at paravelsystems.com (Fred Gleason)
Date: Thu, 21 Mar 2024 12:53:02 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
 <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>
 <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>
Message-ID: <10B6D695-5032-4E5F-A3DE-CFCD4F70DFE4@paravelsystems.com>

On Mar 21, 2024, at 9:02?AM, gARetH baBB <hick.icecast at gink.org> wrote:

> Or you could just use ffmpeg:

Since we?re discussing HLS encoder options, there is also:

	https://github.com/ElvishArtisan/GlassCoder

Which does not require LiquidSoap, ffmpeg or even Icecast. Just aim it at a web server or CDN publishing point.

Cheers!


|---------------------------------------------------------------------|
| Frederick F. Gleason, Jr. |             Chief Developer             |
|                           |             Paravel Systems             |
|---------------------------------------------------------------------|
| But in our enthusiasm, we could not resist a radical overhaul of    |
| the system, in which all of its major weaknesses have been exposed, |
| analyzed, and replaced with new weaknesses.                         |
|                                                                     |
|                     -- Bruce Leverett                               |
|                     "Register Allocation in Optimizing Compilers"   |
|---------------------------------------------------------------------|
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20240321/171c7bac/attachment.htm>

From wayne at cffcs.com  Fri Mar 22 15:05:18 2024
From: wayne at cffcs.com (Wayne Barron)
Date: Fri, 22 Mar 2024 11:05:18 -0400
Subject: [Icecast] Education - 1, 000s, 100, 000's,
 Millions of listeners. (What kind of infrastructure)
In-Reply-To: <10B6D695-5032-4E5F-A3DE-CFCD4F70DFE4@paravelsystems.com>
References: <CAJatk5kGD=Be5GqPDJvLo2ReiR0fpubX-ck3s11FGKr+GoAnLQ@mail.gmail.com>
 <29862364-6202-4C52-83C2-71BCA935BAF7@paravelsystems.com>
 <016101da7b08$3ff6e1b0$bfe4a510$@gmail.com>
 <CAJatk5=zT53KUP6v1VVW_iimrAS5T1gu_oCMVpwYNio=XKDGiA@mail.gmail.com>
 <00ab01da7b66$f7712900$e6537b00$@gmail.com>
 <CAJatk5=BDyMF7ZWwGH4=NU1NcXZ7oqi2kue1hzzQ-UQ3wr6Z-w@mail.gmail.com>
 <5d9c0495-c87a-d639-d383-6f348b07a0b7@gink.org>
 <10B6D695-5032-4E5F-A3DE-CFCD4F70DFE4@paravelsystems.com>
Message-ID: <CAJatk5n_QjTAGq425TH46L-F6BX1AZq3dGoM9_272sTTTjkDJA@mail.gmail.com>

I just read back over what you wrote, Thomas, about the CDN.
(This is just a scenario thought)
In the case of a radio station, I am thinking all the servers would be
run on our end, and all the media files would be hosted on a CDN
somewhere.
Would that be the correct way to do it?
This way, the media files, being the larger load, will be stored in
places they need to be in order for the client to get the stream
faster while the servers are sitting, currently, in my location where
only the website itself runs from.

Would this be the correct thought for this type of scenario?

On Thu, Mar 21, 2024 at 12:53?PM Fred Gleason <fredg at paravelsystems.com> wrote:
>
> On Mar 21, 2024, at 9:02?AM, gARetH baBB <hick.icecast at gink.org> wrote:
>
> Or you could just use ffmpeg:
>
>
> Since we?re discussing HLS encoder options, there is also:
>
> https://github.com/ElvishArtisan/GlassCoder
>
> Which does not require LiquidSoap, ffmpeg or even Icecast. Just aim it at a web server or CDN publishing point.
>
> Cheers!
>
>
> |---------------------------------------------------------------------|
>
> | Frederick F. Gleason, Jr. |             Chief Developer             |
>
> |                           |             Paravel Systems             |
>
> |---------------------------------------------------------------------|
>
> | But in our enthusiasm, we could not resist a radical overhaul of    |
>
> | the system, in which all of its major weaknesses have been exposed, |
>
> | analyzed, and replaced with new weaknesses.                         |
>
> |                                                                     |
>
> |                     -- Bruce Leverett                               |
>
> |                     "Register Allocation in Optimizing Compilers"   |
>
> |---------------------------------------------------------------------|
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast


From talldarkandstrange at icloud.com  Fri Mar 22 20:32:09 2024
From: talldarkandstrange at icloud.com (TDAS)
Date: Fri, 22 Mar 2024 20:32:09 +0000
Subject: [Icecast] Reverse Proxy
In-Reply-To: <084BC1D3-A67E-443F-9AB7-FFC5E5FEE43B@icloud.com>
References: <084BC1D3-A67E-443F-9AB7-FFC5E5FEE43B@icloud.com>
Message-ID: <D139C9EB-BF8B-4A9D-B40D-42360A4700E0@icloud.com>

Anyone???

> On 19 Mar 2024, at 18:41, TDAS <talldarkandstrange at icloud.com> wrote:
> 
> Hi folks
> 
> What's the 2024 line of thinking about reverse proxies for icecast?
> 
> Not talking massive loads like the 1Gbs discussion I saw the other day (man, how I'd like to have those kind of issues!) :) 
> 
> Specifically, Traefik handling https for a few icecast URLs, each with their own docker container? 
> 
> Rick



From railgun.michael at gmail.com  Fri Mar 22 20:55:40 2024
From: railgun.michael at gmail.com (Railgun)
Date: Fri, 22 Mar 2024 21:55:40 +0100
Subject: [Icecast] Reverse Proxy
In-Reply-To: <D139C9EB-BF8B-4A9D-B40D-42360A4700E0@icloud.com>
References: <084BC1D3-A67E-443F-9AB7-FFC5E5FEE43B@icloud.com>
 <D139C9EB-BF8B-4A9D-B40D-42360A4700E0@icloud.com>
Message-ID: <cc3e6cc4-bce6-4389-9b8c-da8ec6536428@gmail.com>

my icecast instlation is behind traefix. and works so far, the only 
problem is on icecast status page each listener has the switch ip adress 
from traefix and not the individual user one.

Am 22.03.2024 um 21:32 schrieb TDAS:
> Anyone???
>
>> On 19 Mar 2024, at 18:41, TDAS <talldarkandstrange at icloud.com> wrote:
>>
>> Hi folks
>>
>> What's the 2024 line of thinking about reverse proxies for icecast?
>>
>> Not talking massive loads like the 1Gbs discussion I saw the other day (man, how I'd like to have those kind of issues!) :)
>>
>> Specifically, Traefik handling https for a few icecast URLs, each with their own docker container?
>>
>> Rick
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast