[Icecast-dev] Fwd: Segmentation fault issue with new install
Jan-Pieter
jplelystad at gmail.com
Thu Apr 30 08:04:33 PDT 2015
Now with attachment ;-)
2015-04-29 8:57 GMT+02:00 "Thomas B. Rücker" <thomas at ruecker.fi>:
> On 04/28/2015 02:27 PM, Jan-Pieter wrote:
> > Hi all,
> > see my reply inline.
> > Cheers, Jan-Pieter
> >
> >
> > 2015-04-26 7:23 GMT+02:00 "Thomas B. Rücker" <thomas at ruecker.fi
> > <mailto:thomas at ruecker.fi>>:
> >
> > On 04/25/2015 07:32 PM, Jan-Pieter wrote:
> > > Hi all,
> > >
> > > I am a long-time user of Icecast, always without problems. However
> i
> > > made a new setup using icecast 2.4.1 on a fresh and quick CentOS
> > VPS.
> >
> > EPEL package or from somewhere else?
> >
> > it's just a standard CentOS 6.6 release, no EPEL. Distri from hosting
> > provider. [root at h2429770 tmp]# uname -a
> > Linux h2429770.stratoserver.net <http://h2429770.stratoserver.net>
> > 2.6.32-042stab103.6 #1 SMP Wed Jan 21 13:07:39 MSK 2015 x86_64 x86_64
> > x86_64 GNU/Linux
> >
> >
> >
> > > The issue is that starting icecast gives a segmentation fault at
> > > around 80% of all starts. No core file nor a entry in error.log
> > is posted.
> >
> > Enable core dumps using "ulimit -c unlimited". The PWD from where you
> > start Icecast needs to be writeable.
> >
> >
> > Even with ulimit -c unlimited and running from a writable PWD no core
> > is dumped.
>
> Seems it's a bit more involved on Centos6:
> http://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/
> I've tried it successfully on a quick install of centos6 that I've
> compiled 2.4.2 on and am going to try with your config.
> (As you don't use url-auth for sources, 2.4.1 is safe for you)
>
> I followed this steps, but still no core file with this type of segfault.
Other programs which segfaults do produce a core dump however. so bit
strange...
>
> > > In the remaining 20% of the starts icecast start with no problem.
> > >
> > > Contents of the error.log with a succesfull start (20% of
> > icecast starts):
> > > [2015-04-25 21:08:40] INFO main/main Icecast 2.4.1 server started
> > > [2015-04-25 21:08:40] INFO connection/get_ssl_certificate No SSL
> > > capability
> > > [2015-04-25 21:08:40] INFO stats/_stats_thread stats thread
> started
> > > [2015-04-25 21:08:40] INFO yp/yp_update_thread YP update thread
> > started
> > > [2015-04-25 21:08:40] INFO source/source_fallback_file mountpoint
> > > /geen-uitzending.mp3 is reserved
> > > [2015-04-25 21:08:40] WARN format/format_get_type Unsupported or
> > > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > > handler for best effort.
> > > [2015-04-25 21:08:40] WARN source/source_fallback_file mountpoint
> > > "/geen-uitzending.mp3" already reserved
> > > [2015-04-25 21:08:40] INFO source/source_main listener count on
> > > /geen-uitzending.mp3 now 0
> > >
> > >
> > > Contents of error.log with a segmentation fault direct after start
> > > (80% of icecast starts):
> > > 2015-04-25 21:09:05] INFO main/main Icecast 2.4.1 server started
> > > [2015-04-25 21:09:05] INFO connection/get_ssl_certificate No SSL
> > > capability
> > > [2015-04-25 21:09:05] INFO yp/yp_update_thread YP update thread
> > started
> > > [2015-04-25 21:09:05] INFO source/source_fallback_file mountpoint
> > > /geen-uitzending.mp3 is reserved
> > > [2015-04-25 21:09:05] WARN format/format_get_type Unsupported or
> > > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > > handler for best effort.
> > > [2015-04-25 21:09:05] INFO stats/_stats_thread stats thread
> started
> >
> > Can you attach your icecast.xml please (with passwords removed)?
> > This might help reproduce the problem.
> >
> >
> > I attached icecast.xml. I also turned debug errorlog level on.
> > Results with debug on:
> >
> > good start:
> > [2015-04-28 15:56:39] INFO main/main Icecast 2.4.1 server started
> > [2015-04-28 15:56:39] DBUG yp/yp_recheck_config Updating YP
> configuration
> > [2015-04-28 15:56:39] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-28 15:56:39] INFO stats/_stats_thread stats thread started
> > [2015-04-28 15:56:39] INFO yp/yp_update_thread YP update thread started
> > [2015-04-28 15:56:39] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-28 15:56:39] DBUG connection/connection_complete_source
> > sources count is 0
> > [2015-04-28 15:56:39] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > handler for best effort.
> > [2015-04-28 15:56:39] DBUG connection/connection_complete_source
> > source is ready to start
> > [2015-04-28 15:56:39] DBUG source/source_init Source creation complete
> > [2015-04-28 15:56:39] WARN source/source_fallback_file mountpoint
> > "/geen-uitzending.mp3" already reserved
> > [2015-04-28 15:56:39] INFO source/source_main listener count on
> > /geen-uitzending.mp3 now 0
> > [2015-04-28 15:56:39] DBUG stats/modify_node_event update global
> > sources (1)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new source stat
> > /geen-uitzending.mp3
> > [2015-04-28 15:56:39] DBUG stats/modify_node_event update global
> > source_client_connections (1)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > listeners (0)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > listenurl (http://stream3.server7.nl:80/geen-uitzending.mp3)
> > [2015-04-28 15:56:39] DBUG stats/modify_node_event update global
> > source_total_connections (1)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > slow_listeners (0)
> > [2015-04-28 15:56:39] DBUG stats/modify_node_event update
> > "/geen-uitzending.mp3" listeners (0)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > listener_peak (0)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > stream_start (Tue, 28 Apr 2015 15:56:39 +0200)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > stream_start_iso8601 (2015-04-28T15:56:39+0200)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > total_bytes_read (0)
> > [2015-04-28 15:56:39] DBUG stats/process_source_event new node
> > total_bytes_sent (0)
> > [2015-04-28 15:56:39] DBUG stats/modify_node_event update
> > "/geen-uitzending.mp3" listeners (0)
> > [2015-04-28 15:56:40] DBUG slave/_slave_thread checking master stream
> > list
>
> I currently suspect it's something interacting with the stats thread and
> the two identical fallbacks.
>
> Agree, seems to have something to do with stats thread. But I tested also
with dual fall back files. Same issue. See attached new configfile.
>
> >
> > bad start (segfault):
> > [2015-04-28 15:56:53] INFO main/main Icecast 2.4.1 server started
> > [2015-04-28 15:56:53] DBUG yp/yp_recheck_config Updating YP
> configuration
> > [2015-04-28 15:56:53] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-28 15:56:53] INFO yp/yp_update_thread YP update thread started
> > [2015-04-28 15:56:53] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-28 15:56:53] DBUG connection/connection_complete_source
> > sources count is 0
> > [2015-04-28 15:56:53] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > handler for best effort.
> > [2015-04-28 15:56:53] DBUG connection/connection_complete_source
> > source is ready to start
> > [2015-04-28 15:56:53] DBUG source/source_init Source creation complete
> > [2015-04-28 15:56:53] INFO stats/_stats_thread stats thread started
> > [2015-04-28 15:56:53] DBUG stats/process_source_event new source stat
> > /geen-uitzending.mp3
> >
> >
> >
> >
> >
> >
> > > Both the stream and the fallback file are .mp3 type. I tried
> > removing
> > > the fallback entry. Then icecast starts always normally. So it
> seems
> > > related to the .mp3 fallback.
> > >
> > > Also I tried to launch icecast with strace, and this NEVER fails.
> No
> > > segmentation faults!
> > >
> > >
> > > Any idea what's going wrong? Something going wrong with timing on a
> > > fast server?
> >
> > It sounds like you've found a race condition. I'm not exactly sure
> > what's happening though.
> > By the log output something seems to be instantly accessing the
> > stream.
> > Notice the order of the stats_thread is different in both.
> >
> >
> > Yes this is the only interesting thing. The segfault only happens when
> > "INFO stats/_stats_thread stats thread started"
> > is missing in the error.log. So with every succesful start this line
> > gets logged, and if the line is missing the segfault occurs.
>
> I suspect it's a very close race that might only occur on that specific
> configuration/virtualization.
> In such case invasive things like strace and valgrind won't help us. A
> coredump might reveal something, but isn't guaranteed.
>
> What BTW confuses me is that you override the mountpoint configuration
> to ogg/vorbis, but fall back to an mp3 file.
>
> Sorry my mistake. Tried changing things here, but did not change the
behaviour. In the attached config file I set it all to mp3 and problem
persists.
Agree that it has probably something to do with virtualization platform. My
VPS is running in a Virtuozzo container. However with other programs no
issue at all. Looks to me still as a timing issue that somehow the stats
thread is not started causing the segfault when loading the fallback
stream. Maybe a small noop/sleep just before the statsthread will solve my
issue...
> > I also tried running using valgrind on advice of Philipp, but when
> > running with valgrind the segfault does not occur and icecast starts
> > normally... (same as with gdb)
>
> yeah, expected.
>
> I tried your config on a centos 6 machine and couldn't reproduce it. How
> many CPU cores / threads does it have?
>
> 4 cores/threads.
>
> Cheers
>
> Thomas
>
> Thanks!
Jan-Pieter
> _______________________________________________
> Icecast-dev mailing list
> Icecast-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/icecast-dev/attachments/20150430/373a1a34/attachment-0001.htm
-------------- next part --------------
Here you see it not always fails 80% of all starts. Sometimes 6x in a row segfault, sometimes 3x in a row succesful start.
Server runs with 4 cores
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [ OK ]
Icecast Streaming-Daemon starten: [ OK ]
[root at h2429770 etc]# service icecast restart
Icecast-Streaming-Daemon beenden: [FAILED]
Icecast Streaming-Daemon starten: [ OK ]
contents of icecast.xml after changing to independent fallback mount files:
[root at h2429770 etc]# cat icecast.xml
<icecast>
<!-- location and admin are two arbitrary strings that are e.g. visible
on the server info page of the icecast web interface
(server_version.xsl). -->
<location>Lelystad</location>
<admin>icecaststreams at server7.nl</admin>
<!-- IMPORTANT!
Especially for inexperienced users:
Start out by ONLY changing all passwords and restarting Icecast.
For detailed setup instructions please refer to the documentation.
It's also available here: http://icecast.org/docs/
-->
<limits>
<clients>100</clients>
<sources>2</sources>
<queue-size>524288</queue-size>
<client-timeout>30</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>10</source-timeout>
<!-- If enabled, this will provide a burst of data when a client
first connects, thereby significantly reducing the startup
time for listeners that do substantial buffering. However,
it also significantly increases latency between the source
client and listening client. For low-latency setups, you
might want to disable this. -->
<burst-on-connect>1</burst-on-connect>
<!-- same as burst-on-connect, but this allows for being more
specific on how much to burst. Most people won't need to
change from the default 64k. Applies to all mountpoints -->
<burst-size>65535</burst-size>
</limits>
<authentication>
<!-- Sources log in with username 'source' -->
<source-password>pass</source-password>
<!-- Relays log in with username 'relay' -->
<relay-password>pass</relay-password>
<!-- Admin logs in with the username given below -->
<admin-user>admin</admin-user>
<admin-password>pass</admin-password>
</authentication>
<!-- set the mountpoint for a shoutcast source to use, the default if not
specified is /stream but you can change it here if an alternative is
wanted or an extension is required
<shoutcast-mount>/live.nsv</shoutcast-mount>
-->
<!-- Uncomment this if you want directory listings -->
<!--
<directory>
<yp-url-timeout>15</yp-url-timeout>
<yp-url>http://dir.xiph.org/cgi-bin/yp-cgi</yp-url>
</directory>
-->
<!-- This is the hostname other people will use to connect to your server.
It affects mainly the urls generated by Icecast for playlists and yp
listings. You MUST configure it properly for YP listings to work!
-->
<hostname>stream3.server7.nl</hostname>
<!-- You may have multiple <listener> elements -->
<listen-socket>
<port>80</port>
<bind-address>85.214.159.197</bind-address>
<!-- <shoutcast-mount>/stream</shoutcast-mount> -->
</listen-socket>
<listen-socket>
<port>8000</port>
</listen-socket>
<listen-socket>
<port>7999</port>
</listen-socket>
<!--
<listen-socket>
<port>8443</port>
<ssl>1</ssl>
</listen-socket>
-->
<!-- Global header settings
Headers defined here will be returned for every HTTP request to Icecast.
The ACAO header makes Icecast public content/API by default
This will make streams easier embeddable (some HTML5 functionality needs it).
Also it allows direct access to e.g. /status-json.xsl from other sites.
If you don't want this, comment out the following line or read up on CORS.
-->
<http-headers>
<header name="Access-Control-Allow-Origin" value="*" />
</http-headers>
<!-- Relaying
You don't need this if you only have one server.
Please refer to the config for a detailed explanation.
-->
<!--<master-server>127.0.0.1</master-server>-->
<!--<master-server-port>8001</master-server-port>-->
<!--<master-update-interval>120</master-update-interval>-->
<!--<master-password>hackme</master-password>-->
<!-- setting this makes all relays on-demand unless overridden, this is
useful for master relays which do not have <relay> definitions here.
The default is 0 -->
<!--<relays-on-demand>1</relays-on-demand>-->
<!--
<relay>
<server>127.0.0.1</server>
<port>8080</port>
<mount>/example.ogg</mount>
<local-mount>/different.ogg</local-mount>
<on-demand>0</on-demand>
<relay-shoutcast-metadata>0</relay-shoutcast-metadata>
</relay>
-->
<!-- Mountpoints
Only define <mount> sections if you want to use advanced options,
like alternative usernames or passwords
-->
<!-- Default settings for all mounts that don't have a specific <mount type="normal">.
-->
<!--
<mount type="default">
<public>0</public>
<intro>/server-wide-intro.ogg</intro>
<max-listener-duration>3600</max-listener-duration>
<authentication type="url">
<option name="mount_add" value="http://auth.example.org/stream_start.php"/>
</authentication>
<http-headers>
<header name="foo" value="bar" />
</http-headers>
</mount>
-->
<!-- Normal mounts -->
<mount type="normal">
<mount-name>/lichtbron</mount-name>
<username></username>
<password>pass</password>
<max-listeners>100</max-listeners>
<max-listener-duration>14400</max-listener-duration>
<dump-file>/tmp/dumplichtbron.mp3</dump-file>
<intro></intro>
<fallback-mount>/geen-uitzending1.mp3</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<charset>ISO8859-1</charset>
<public>1</public>
<stream-name>GKV Lichtbron</stream-name>
<stream-description>1Live vanuit de Lichtbron in Lelystad</stream-description>
<stream-url>http://www.pauwenburg.nl/beamer/beamerhq.html</stream-url>
<genre>kerkdienst</genre>
<bitrate>128</bitrate>
<type>application/mp3</type>
<subtype>mp3</subtype>
<hidden>0</hidden>
<burst-size>65536</burst-size>
<mp3-metadata-interval>4096</mp3-metadata-interval>
</mount>
<mount type="normal">
<mount-name>/pijlerjp</mount-name>
<username></username>
<password>pass</password>
<max-listeners>25</max-listeners>
<max-listener-duration>14400</max-listener-duration>
<dump-file>/var/www/vhosts/pijlervideo.server7.nl/httpdocs/Audio/dienst.mp3</dump-file>
<intro></intro>
<fallback-mount>/geen-uitzending2.mp3</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<charset>ISO8859-1</charset>
<public>1</public>
<stream-name>TestStream Pijler</stream-name>
<stream-description>Live teststream Pijler - info jpveen</stream-description>
<stream-url>http://www.pijler.nl/</stream-url>
<genre>test</genre>
<bitrate>128</bitrate>
<type>application/mp3</type>
<subtype>mp3</subtype>
<hidden>0</hidden>
<burst-size>65536</burst-size>
<mp3-metadata-interval>4096</mp3-metadata-interval>
</mount>
<!--
<mount type="normal">
<mount-name>/example-complex.ogg</mount-name>
<username>othersource</username>
<password>hackmemore</password>
<max-listeners>1</max-listeners>
<dump-file>/tmp/dump-example1.ogg</dump-file>
<burst-size>65536</burst-size>
<fallback-mount>/example2.ogg</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>1</fallback-when-full>
<intro>/example_intro.ogg</intro>
<hidden>1</hidden>
<public>1</public>
<authentication type="htpasswd">
<option name="filename" value="myauth"/>
<option name="allow_duplicate_users" value="0"/>
</authentication>
<http-headers>
<header name="Access-Control-Allow-Origin" value="http://webplayer.example.org" />
<header name="baz" value="quux" />
</http-headers>
<on-connect>/home/icecast/bin/stream-start</on-connect>
<on-disconnect>/home/icecast/bin/stream-stop</on-disconnect>
</mount>
-->
<!--
<mount type="normal">
<mount-name>/auth_example.ogg</mount-name>
<authentication type="url">
<option name="mount_add" value="http://myauthserver.net/notify_mount.php"/>
<option name="mount_remove" value="http://myauthserver.net/notify_mount.php"/>
<option name="listener_add" value="http://myauthserver.net/notify_listener.php"/>
<option name="listener_remove" value="http://myauthserver.net/notify_listener.php"/>
<option name="headers" value="x-pragma,x-token"/>
<option name="header_prefix" value="ClientHeader."/>
</authentication>
</mount>
-->
<fileserve>1</fileserve>
<paths>
<!-- basedir is only used if chroot is enabled -->
<basedir>/opt/icecast/latest/share/icecast</basedir>
<!-- Note that if <chroot> is turned on below, these paths must both
be relative to the new root, not the original root -->
<!-- <logdir>/opt/icecast/latest/var/log/icecast</logdir> -->
<webroot>/opt/icecast/latest/share/icecast/web</webroot>
<adminroot>/opt/icecast/latest/share/icecast/admin</adminroot>
<!-- <pidfile>/opt/icecast/latest/share/icecast/icecast.pid</pidfile> -->
<logdir>/var/log/icecast</logdir>
<pidfile>/var/run/icecast/icecast.pid</pidfile>
<!-- Aliases: treat requests for 'source' path as being for 'dest' path
May be made specific to a port or bound address using the "port"
and "bind-address" attributes.
-->
<!--
<alias source="/foo" destination="/bar"/>
-->
<!-- Aliases: can also be used for simple redirections as well,
this example will redirect all requests for http://server:port/ to
the status page
<alias source="/" destination="/status.xsl"/>
-->
<alias source="/" dest="/secure.xsl"/>
<alias source="/jp" dest="/statusjp.xsl"/>
<!-- The certificate file needs to contain both public and private part.
Both should be PEM encoded.
<ssl-certificate>/opt/icecast/latest/share/icecast/icecast.pem</ssl-certificate>
-->
</paths>
<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<!-- <playlistlog>playlist.log</playlistlog> -->
<loglevel>4</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
<logsize>10000</logsize> <!-- Max size of a logfile -->
<!-- If logarchive is enabled (1), then when logsize is reached
the logfile will be moved to [error|access|playlist].log.DATESTAMP,
otherwise it will be moved to [error|access|playlist].log.old.
Default is non-archive mode (i.e. overwrite)
-->
<!-- <logarchive>1</logarchive> -->
</logging>
<security>
<chroot>0</chroot>
<changeowner>
<user>icecast</user>
<group>icecast</group>
</changeowner>
</security>
</icecast>
[root at h2429770 etc]#
More information about the Icecast-dev
mailing list