[Icecast] Icecast 2.3.3 + Debian

[Rémi Cardona]

Le 08/07/2012 21:19, Rücker Thomas a écrit :
> Why have you not filed bugs about it then? Outlining the gravity of the
> problem?
> Or at least brought it up on IRC.

The biggest reason is that the network ops at my company don't report
many of the crashes/bugs to us (I'm on the dev side). They install monit
[1] on all the servers and have it restart icecast as soon as something
goes wrong. So up until recently, I didn't think the situation was as
bad as it really is.

The second reason was this race work done done outside our regular
assignments so we had no authorization to put this on production servers...

> I suspect you mean
> https://trac.xiph.org/ticket/1810

Indeed.

> The ticket does not suggest that those are in any way proven and grave
> issues.
> It merely suggests them as 'potential' issues.
[...]
> As such it didn't make it into 2.3.3 and was deferred for 2.4.
> If you have proof that those race conditions occur on your systems it
> would have
> been beneficial to know that earlier. We are of course interested in
> Icecast being
> as stable as possible.

The thing is, finding proof and factual evidence of race conditions is
one of the hardest thing one can do in any programming language. So far,
the most common "proof" I've seen on production servers are double
free() (which trigger an abort() on Linux), listener counters not
incrementing/decrementing properly and socket leaks. All those issues
are almost impossible to reproduce, which is what lead us to the race
condition theory and to run icecast in Hellgrind.

Hellgrind is a god-send as it emulates a CPU and records (almost) all
memory operations and analyzes if/how locks are taken. So apart from a
few false-positives (though I haven't seen any while working on
icecast), all the errors reported by Hellgrind are indeed race
conditions waiting to happen.

FTR, Laurent and I only tested those patches on our own machines. Those
patches could impact icecast's performance significantly. Consider them
a work in progress.

Cheers,

Rémi

[1] http://mmonit.com/monit/