[Icecast-dev] yp.xml - Size and Contents

[Rücker Thomas]

Hi everyone,

On 10/07/12 18:52, Assen Totin wrote:
> 1. I recently noticed that yp.xml is still limited to 1,000 entries 
> only while the web directory says there are over 21K streams. I 
> remember some discussions regarding the bandwidth required to serve 
> this file, so is it possible to get it in compressed format? Here are 
> some figures:
>
> - the yp.xml as served now (1K entries) is 343 KB. This means 21K 
> entries will be around 7 MB.
>
> - compressing yp.xml with gzip (default options on Linux, not even 
> -9)  gives a yp.xml.gz of 14 KB. This means 21K compressed entries 
> will be less than 300 KB, or less than the current uncompressed 1K! 
> gzip is fast enough to be used on-the-fly (bzip2 gives even smaller 
> size, around 8 KB for 1K entries, but is an order of magnitude 
> slower). If CPU usage is a concern, then to avoid on-the-fly 
> compression, yp.xml can be compressed, say, once every minute, cached 
> and served from the cache.

We're aware of those issues and as much as we wanted to see this 
resolved earlier it was not possible as we rely on server and foremost 
bandwidth capacity provided to us for free.
I don't have the time to venture into details right now, but just to 
give you an idea of the scale of this unusual setup:
- hundreds of hits per second to apache
- hundreds of queries per second to the database
- transient database in RAM
- egress of about 2Megabit per second, 24/7

That said we're currently working on moving hosting of dir.xiph.org back 
to xiph servers as we have now the capacity for this and also want to 
resolve the problem with the limit. Along the way we'll most likely 
implement further changes to the service. If you are maintaining 
software that accesses dir.xiph.org/yp.xml - please stay tuned for updates!

> 2. Also, I noticed at hat all 1K entries in yp.xml right now come from 
> the same web site, radionomy.com <http://radionomy.com>. It claims ti 
> has over 3K streams. However, there are some issues with their entries 
> which make them nearly useless:
>
> - all of their streams are listed with the "server_name" tag set to 
> "Unspecified name". Having 3,000 stations named "Unspecified name" is 
> not very useful. Every station at radionomy.com <http://radionomy.com> 
> has a distinguished "name" which is actually part of the URL - could 
> they send this actual name as "server_name"?
>
> - all of their streams have the "genre" tag set to "various". Again, 
> not very helpful; their web site says there are clearly different 
> genres, about a dozen, and each station belongs to one; why cannot 
> they send the proper genre name?
>
> The only email I could find on their web site is faq at radionomy.com 
> <mailto:faq at radionomy.com>, all other contact info is in the form of 
> web forms.
That is indeed 'fascinating'.

We'll have to have a closer look after the transition. Thanks for 
bringing this up!

Cheers

Thomas