[Icecast] dir.xiph.org only listing Ogg streams

[Ralph Giles]

On Mon, Jun 19, 2006 at 12:54:19PM +0100, Karl Heyes wrote:

> I really would like to know what the effect was with the around 1100 
> entries. I just cannot imagine the bandwidth being the issue, not with 
> updates over 2 mins intervals. 1kB x 1100 is only 1MB over 2 mins is not 
> a bandwidth hog. I suspect the issue is more to do with apache going 
> thread/process crazy, the sql DB, and/or reconnecting to the DB all the 
> time.

Sadly, it's not the php, because it would have been easy to replace the 
frontpage with a static view updated every couple of minutes. It was the 
updates from the icecast instances that was killing us.

With 1100 servers listed we were doing 100 mysql queries/second and 
about 12 kB/s data transfer. Now with 250 servers we're doing 22 
queries/second and ~2 kB/s.

Basically the database was getting disk bound, I think. We never ran out 
of CPU or network bandwidth, but load average would spike sometimes and 
we kept getting into swap even with 3GB of ram. Tuning the db 
design in update method might help.

 -r