[Icecast] Source Drops, metadata, and source limits in setup

Tue Jul 18 10:22:39 UTC 2017

Good morning,

On Sat, 2017-07-15 at 20:51 +0000, ScanCaster wrote:
> On Fri, 14 Jul 2017 09:16:34 +0000, Philipp Schafft wrote:
> 
> > This posting confirms me in that I should write a separate e-mail to the
> > list on how metadata updates work and how they should be implemented on
> > the source side.
> 
> And what is different in the way this is updated???

The difference is that doing it correctly does not break the data stream
and will not cause random failure of whatever component in the line.

> > Why don't you use -O /dev/null?
> 
> -O or > /dev/null does the same thing. Like many things in Linux 40 ways 
> to do it.

You're not using -O - at all. And it does not. --delete-after does
something totally different.

> > Also make sure you both quote the song
> > parameter first to HTTP standards and then to shell standards.
> 
> The BASH script takes in this: "039000DB13BE00DB13BE189F" decodes it and 
> spits out:
> [This+is+who+is+talking] 

I don't see the correlation to what I said.

> > I think the redirect of stdout is mostly useless. This also will allow
> > each and every process on the same system to spot the password.
> 
> Since its my box, or my own little devices on the network, and I am the 
> only one who has access its not an issue.
> 
> Each Source gets a unique password. 

True until it changes. Sounds like very fragile security to me.

> > Which *exact* version of Icecast2 do you run and where have you got it
> > from? Which operating system are you using?
> 
> IceCast V2.4.1 on Ubuntu 14.04 on a VPS with 2GB RAM.

ok.

> > Icecast2 will write a message to the error.log in case there is some
> > problem with a client. What is the relevant section of the error.log?
> > Also have a look at the access.log. It will list all the clients after
> > they have disconnected. This includes the source connections as well as
> > admin interface interaction.
> > 
> > Without input from the log files it's hard to tell. Here are some of my
> > guesses:
> >       * Your network has some kind of a problem. Maybe a bad cable right
> >         on the server? (Would explain why source fail at the same time
> >         and why altering timeouts may help.)
> 
> Nope. Internet at source continues to operate fine. Live data coming on 
> the network to verify it is running.

The logs you provided below strongly indicate a network related problem
OR a broken encoder setup. It could also trying to encode digital
silence with a 21th century codec (Vorbis, Opus, ...).

> >       * You're running a non-official version of Icecast2. (could
> >         explain everything or nothing.)
> 
> Nope, V2.4.1 from Ubuntu 14.04 repo

Ok.

> >       * One of your scripts do random things. E.g. because of faulty
> >         escaping. (Could explain random drops.)
> 
> The metadata is "escaped" for BASH. Its straight plain text.

It needs to be escaped for both HTTP and your shell. Otherwise it could
easily break the HTTP part. e.g. if it contains am ampersand ('&').

> >       * Your admin and/or source credentials are known to someone else.
> >         (Could explain all of those things.)
> 
> Unlikely.
> 
> >       * You're not running one of the stable and supported versions.
> >         There could be an attack based on a security problem of an old
> >         version going on. (Could explain all of those things.)
> 
> V2.4.1 is the version I have to run, as a change in the code for metadata 
> updates checks the source IP.. We have reasons for running metadaa 
> updates on sources which the script on the server will not be from the 
> sources IP.

This is simply not correct.
The limitation only applies for raw requests with source credentials. It
is not true for Admin interface requests (rendered) or requests with
admin credentials. And considering your '--http-user=admin' from the
code you have you're likely using admin credentials. (I don't think you
reconfigured Icecast so the source username is 'admin'.)

> >       * Your machine has some kind of hardware malfunction, e.g. bad
> >         RAM. (Could explain all of those things.)
> 
> I've have similar issues on 3 differing hosts over numerous years. From 2 
> VPS on CentOS to another on Ubuntu... the hosts had other issues not 
> related to why I changed.

Which brings us back to my first point as it would mean hardware or
operating system is likely not the cause.

> >       * A still unknown bug in Icecast2. (Could explain all of those
> >         things.)
> 
> 
> >       * A mixture of what is above.
> > 
> > The logs will easily tell if there is a timeout related problem, that
> > always means the network has some problem. It will also tell if there
> > are related admin interface requests. Both of this considering one of
> > the stable and supported versions of Icecast2.
> 
> 
> This is IceCast V 2.4.1 from the Ubuntu 14.04 on a OVZ VPS with 2GB RAM.
> 
> Please note, changing from 2.4.1 to any other version is NOT POSSIBLE. 
> Unless of course the code change that blocks the IP check on metadata 
> updates has been changed to allow any IP to update it.

See above.

> While that may be a security issue by definition for some...
> In my form of broadcasting the server has several server wide
> metadata updates that are applied server wide or for a number
> feeds based on triggers. 

It is. And it is not a problem for streams doing metadata correctly to
begin with.

> From error.log"
> 
> 2017-07-15  15:43:04] WARN source/get_next_buffer Disconnecting source 
> due to socket timeout

'socket timeout' clearly means, that Icecast2 is unable to read data
from the source client for whatever reason (network failure or the
source client not sending data for whatever reasons).

> [2017-07-15  15:43:04] INFO source/source_shutdown Source from 
> 255.255.255.255 at "/mount" exiting

> That line repeats...for each as it happens.. This is the first time in 4 
> days.

yes.

> Sporadic internet blips, happen. They can't all happen at the same time.

Sure they can. e.g. if it's close by. A common example of this a network
cable not correctly plugged into the server. Have seen errors like that
many times. There is a reason why the first question of the helpdesk
always is 'make sure all cables are plugged in correctly'.

> Then the source from location 50 miles away AND different ISP, same 
> things.
> 
> The source(s) reconnect:
> 
> [2017-07-15  15:43:45] INFO connection/_handle_source_request Source 
> logging in at mountpoint "/mount" from 255.255.255.255
> [2017-07-15  15:43:45] WARN format/format_get_type Unsupported or legacy 
> stream type: "audio/mpeg". Falling back to generic minimal handler for 
> best effort.
> 
> Sources are provided by Linux boxes using DarkIce, except the ONE foreign 
> IP one, which has some winslopper application, which is being replaced 
> with one of my dedicated broadcasting boxes on Linux the first week of 
> August. 
> 
> access.log just shows the DarkIce client/sources reconnecting.
> 
>   [15/Jul/2017:15:43:52 -0400] "SOURCE /PL HTTP/1.0" 200 19 "-" 
> "DarkIce/0.20.1 (http://darkice.tyrell.hu/)" 317316
>  [15/Jul/2017:15:43:53 -0400] "GET /admin/metadata HTTP/1.0" 401 331 "-" 
> "Wget/1.12 (linux-gnu)" 0
>  [15/Jul/2017:15:43:53 -0400] "GET /admin/metadata HTTP/1.0" 400 342 "-" 
> "Wget/1.12 (linux-gnu)" 0
>  [15/Jul/2017:15:43:56 -0400] "SOURCE /H.ogg HTTP/1.0" 200 19 "-" 
> "DarkIce/0.20.1 (http://darkice.tyrell.hu/)" 317307
>  [15/Jul/2017:15:44:03 -0400] "SOURCE /Po.ogg HTTP/1.0" 200 19 "-" 
> "DarkIce/0.20.1 (http://darkice.tyrell.hu/)" 263472
> - [15/Jul/2017:15:44:04 -0400] "SOURCE /P HTTP/1.0" 200 19 "-" 
> "DarkIce/1.0 (http://code.google.com/p/darkice/)" 19
>  - - [15/Jul/2017:15:44:05 -0400] "GET /admin/metadata HTTP/1.0" 401 331 
> "-" "Wget/1.12 (linux-gnu)" 0
> 1 - - [15/Jul/2017:15:44:06 -0400] "GET /ma HTTP/1.1" 200 346235 "-" 
> "Softwaree/5" 72
> - - [15/Jul/2017:15:44:08 -0400] "SOURCE /PlaD HTTP/1.0" 200 19 "-" 
> "DarkIce/1.0 (http://code.google.com/p/darkice/)" 0

With best regards,

-- 
Philipp.
 (Rah of PH2)