[Icecast] "Ghost" connections causing connection limit to be reached

[Rob Hailman]

Interesting, thanks Tony.

There are some entries that don't have bot UAs but those of course could be
spoofed.

It's curious that this is only affecting the ogg streams - bots seemingly
are connecting to the mp3 streams but don't get "stuck" like this. Perhaps
the bot is "moving on" in a way that Icecast doesn't notice? In the logs I
periodically see "Client XXXX ([IP address]) has fallen too far behind,
removing" but seemingly only for the mp3 stream - the ogg streams just
accumulate bots until the connection limit is reached.

It does look like Icecast has both those features - with the
<max-listener-duration> and <deny-ip> configuration keys. It seems like
setting a max duration (even a very generous one like 24 hours) will solve
our issue.

Thanks,
Rob

On Fri, Feb 3, 2023 at 4:57 AM Tony Harding <uktony at radiocompany.net> wrote:

> Bots are a Web Crawler problem. They check a web page and follow all the
> links. One is to your stream. They wait for the “Page” to finish loading,
> but as this is a stream, it never finishes loading. They just see data
> coming and do not check if it is audio or HTML.
>
>
>
> Not that I am advocating for Shoutcast, but it does have two features that
> would be useful, A ban list, and a maximum duration setting. No one listens
> for more than 24 hours.
>
>
>
> Tony
>
>
> ------------------------------
>
> *From:* Icecast [mailto:icecast-bounces at xiph.org] *On Behalf Of *Rob
> Hailman
> *Sent:* Friday, February 3, 2023 6:53 AM
> *To:* icecast at xiph.org
> *Subject:* [Icecast] "Ghost" connections causing connection limit to be
> reached
>
>
>
> Hello,
>
>
>
> I've been having an issue periodically where our Icecast server (which is
> set to a max 200 connections) will reach that limit and start rejecting all
> connections, but most of the connections don't seem to be real active
> listeners.
>
>
>
> We are running icecast 2.4.4, installed via apt on Ubuntu 22.04.1 LTS.
>
>
>
> We have four streams - two mp3 and two ogg. It seems like what is
> happening is that connections to the ogg streams are never getting released
> - so the number of clients will go up and up until all 200 connections are
> taken up. An excerpt from error.log, showing the mp3 stream count
> fluctuating but the .ogg count only increasing:
>
>
>
> [2023-01-29  10:25:46] INFO source/source_main listener count on /hi-fi
> now 6
> [2023-01-29  10:25:47] INFO source/source_main listener count on /hi-fi
> now 5
> [2023-01-29  10:25:47] INFO source/source_main listener count on
> /hi-fi.ogg now 124
> [2023-01-29  10:25:51] INFO source/source_main listener count on /hi-fi
> now 4
> [2023-01-29  10:26:17] INFO source/source_main listener count on
> /hi-fi.ogg now 125
> [2023-01-29  10:26:47] INFO source/source_main listener count on
> /hi-fi.ogg now 126
>
>
>
> When I restart icecast, the logs will show all 200 connections being
> released - and in access.log, most of the .ogg connections will be for very
> long durations with very small amounts of data transferred - for example:
>
>
>
> 77.88.9.3 - - [02/Feb/2023:21:40:29 +0000] "GET /hi-fi.ogg HTTP/1.1" 200
> 422 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
> 2694965
>
>
>
> If I'm reading that correctly, the connection was open for 31 days with
> 422 bytes transferred.
>
>
>
> Many (but not all) of these connections seem to be bots.
>
>
>
> Would be happy to provide the full log files but didn't want to send >1MB
> files to the whole list.
>
>
>
> Is this a possible configuration issue, an Icecast bug, or something else?
> Any help or suggestions would be greatly appreciated!
>
>
>
> Thanks,
>
> Rob
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20230203/4e631afd/attachment.htm>