<div dir="auto"><div dir="auto">You could quite simply use the url auth listener_add hook to reject those bots based on the useragent.</div><div dir="auto"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">בתאריך יום ו׳, 3 בפבר׳ 2023, 22:20, מאת Rob Hailman <<a href="mailto:rhailman@trentradio.ca" target="_blank" rel="noreferrer">rhailman@trentradio.ca</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Interesting, thanks Tony.<div><br></div><div>There are some entries that don't have bot UAs but those of course could be spoofed.</div><div><br></div><div>It's curious that this is only affecting the ogg streams - bots seemingly are connecting to the mp3 streams but don't get "stuck" like this. Perhaps the bot is "moving on" in a way that Icecast doesn't notice? In the logs I periodically see "<span style="color:rgb(0,0,0);white-space:pre-wrap">Client XXXX ([IP address]) has fallen too far behind, removing" but seemingly only for the mp3 stream - the ogg streams just accumulate bots until the connection limit is reached.</span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap"><br></span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap">It does look like Icecast has both those features - with the <max-listener-duration> and <deny-ip> configuration keys. It seems like setting a max duration (even a very generous one like 24 hours) will solve our issue.</span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap"><br></span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap">Thanks,</span></div><div><span style="color:rgb(0,0,0);white-space:pre-wrap">Rob</span></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 3, 2023 at 4:57 AM Tony Harding <<a href="mailto:uktony@radiocompany.net" rel="noreferrer noreferrer" target="_blank">uktony@radiocompany.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-GB">
<div>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy">Bots are a Web Crawler problem. They check
a web page and follow all the links. One is to your stream. They wait for the “Page”
to finish loading, but as this is a stream, it never finishes loading. They
just see data coming and do not check if it is audio or HTML.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy">Not that I am advocating for Shoutcast,
but it does have two features that would be useful, A ban list, and a maximum duration
setting. No one listens for more than 24 hours.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy">Tony<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span lang="EN-US" style="font-size:12pt">
<hr size="2" width="100%" align="center">
</span></font></div>
<p class="MsoNormal"><b><font size="2" face="Tahoma"><span lang="EN-US" style="font-size:10pt;font-family:Tahoma;font-weight:bold">From:</span></font></b><font size="2" face="Tahoma"><span lang="EN-US" style="font-size:10pt;font-family:Tahoma">
Icecast [mailto:<a href="mailto:icecast-bounces@xiph.org" rel="noreferrer noreferrer" target="_blank">icecast-bounces@xiph.org</a>] <b><span style="font-weight:bold">On
Behalf Of </span></b>Rob Hailman<br>
<b><span style="font-weight:bold">Sent:</span></b> Friday, February 3, 2023
6:53 AM<br>
<b><span style="font-weight:bold">To:</span></b> <a href="mailto:icecast@xiph.org" rel="noreferrer noreferrer" target="_blank">icecast@xiph.org</a><br>
<b><span style="font-weight:bold">Subject:</span></b> [Icecast]
"Ghost" connections causing connection limit to be reached</span></font><span lang="EN-US"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Hello,<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">I've been having an issue periodically where our Icecast server (which
is set to a max 200 connections) will reach that limit and start rejecting all
connections, but most of the connections don't seem to be real active
listeners.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">We are running icecast 2.4.4, installed via apt on Ubuntu 22.04.1 LTS.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">We have four streams - two mp3 and two ogg. It seems like what is
happening is that connections to the ogg streams are never getting released -
so the number of clients will go up and up until all 200 connections are taken
up. An excerpt from error.log, showing the mp3 stream count fluctuating but the
.ogg count only increasing:<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">[2023-01-29 10:25:46] INFO source/source_main listener count on
/hi-fi now 6<br>
[2023-01-29 10:25:47] INFO source/source_main listener count on /hi-fi
now 5<br>
[2023-01-29 10:25:47] INFO source/source_main listener count on
/hi-fi.ogg now 124<br>
[2023-01-29 10:25:51] INFO source/source_main listener count on /hi-fi
now 4<br>
[2023-01-29 10:26:17] INFO source/source_main listener count on
/hi-fi.ogg now 125<br>
[2023-01-29 10:26:47] INFO source/source_main listener count on
/hi-fi.ogg now 126<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">When I restart icecast, the logs will show all 200 connections being
released - and in access.log, most of the .ogg connections will be for very
long durations with very small amounts of data transferred - for example:<u></u><u></u></span></font></p>
</div>
<div><span style="white-space:pre-wrap">
<p class="MsoNormal"><font size="3" color="black" face="Times New Roman"><span style="font-size:12pt;color:black"><br>
<br>
</span></font><u></u><u></u></p>
</span></div>
<div><span style="white-space:pre-wrap">
<p class="MsoNormal"><font size="3" color="black" face="Times New Roman"><span style="font-size:12pt;color:black">77.88.9.3 - - [02/Feb/2023:21:40:29 +0000]
"GET /hi-fi.ogg HTTP/1.1" 200 422 "-" "Mozilla/5.0
(compatible; YandexBot/3.0; +<a href="http://yandex.com/bots" rel="noreferrer noreferrer" target="_blank">http://yandex.com/bots</a>)"
2694965</span></font></p></span><u></u><u></u>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">If I'm reading that correctly, the connection was open for <font color="black"><span style="color:black"><span style="white-space:pre-wrap">31
days with 422 bytes transferred.</span></span></font><u></u><u></u></span></font></p>
</div>
<div><span style="white-space:pre-wrap">
<p class="MsoNormal"><font size="3" color="black" face="Times New Roman"><span style="font-size:12pt;color:black"><br>
<br>
</span></font><u></u><u></u></p>
</span></div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Many (but not all) of these connections seem to be bots.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Would be happy to provide the full log files but didn't want to send
>1MB files to the whole list.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Is this a possible configuration issue, an Icecast bug, or something
else? Any help or suggestions would be greatly appreciated!<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Thanks,<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12pt">Rob<u></u><u></u></span></font></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Icecast mailing list<br>
<a href="mailto:Icecast@xiph.org" rel="noreferrer noreferrer" target="_blank">Icecast@xiph.org</a><br>
<a href="http://lists.xiph.org/mailman/listinfo/icecast" rel="noreferrer noreferrer noreferrer" target="_blank">http://lists.xiph.org/mailman/listinfo/icecast</a><br>
</div></blockquote></div>
_______________________________________________<br>
Icecast mailing list<br>
<a href="mailto:Icecast@xiph.org" rel="noreferrer noreferrer" target="_blank">Icecast@xiph.org</a><br>
<a href="http://lists.xiph.org/mailman/listinfo/icecast" rel="noreferrer noreferrer noreferrer" target="_blank">http://lists.xiph.org/mailman/listinfo/icecast</a><br>
</blockquote></div></div>