[Icecast] Icecast locks with WARN connection/_accept_connection accept() failed with error 24: Too many open files

[Michael Smith]

On 6/1/05, Joel Ebel <jbebel at ncsu.edu> wrote:
> My icecast server has been running happily for months now, but just
> yesterday it locked up, chewing up all the cpu time it could find, and
> not allowing connections.  The last snippet of the error log shows this:
snip...
> The error had repeated itself for almost 24 hours before I noticed it.
> Obviously it had too many files open, but I can't account for how.  The
> logs before the repeating errors look completely normal.  I tried
> shutting down the streams, which didn't fix it, so I sent it some kills,
> and it didn't die.  I had to kill -9 it.  Once it restarted it was fine.
>   Is there some leak preventing icecast from running indefinitely?  Or
> will I just need to restart it every few months when this happens again?
>   Has anyone else seen this happen?
> 

There's at least one, and possibly two bugs here.

One is that, when the server runs out of file descriptors, from your
report, it looks like it spins - it should just reject
connections/fail gracefully. That one should be straightforward to
fix.

The second problem is either a) you legitimately ran out of file
descriptors (the per-process limit should be easy to change, but I
can't suggest how since you didn't mention your OS), or b) icecast
leaked file descriptors.

b) is more likely, unless you had a lot of clients connected (probably
at least several hundred), and is obviously much worse. You also
didn't mention what version you're running - if it isn't the latest
release, please upgrade.

Was there any strange/suspicious behaviour just before this happened
visible in the logs? If not, it's likely a slow leak caused in some
unusual case, which only becomes a problem after some weeks/months
(but might be exploitable as a DoS attack).

Mike$