Avahi causing mt-daap to segfault

[H3g3m0n]

Binary package hint: mt-daapd

mt-daap starts fine without Avahi running but crashes when it is... Jaunty Alpha 6.

.-(~)---------------------------------------------------------------------------------------------------------------------(hegemon@hextor)-
`--> sudo mt-daapd -f -c /etc/mt-daapd.conf
Firefly Version svn-1696: Starting with debuglevel 2
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-script.so: plugin declined to load
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: undefined symbol: avcodec_decode_audio
Plugin loaded: daap/svn-1696
Plugin loaded: rsp/svn-1696
Starting signal handler
Starting rendezvous daemon
Client running
Initializing database
Starting web server from /usr/share/mt-daapd/admin-root on port 3689
Registering rendezvous names
Client failure
Server disconnected, reconnecting
Client running
zsh: segmentation fault sudo mt-daapd -f -c /etc/mt-daapd.conf
.-(~)---------------------------------------------------------------------------------------------------------------------(hegemon@hextor)-
`--> sudo /etc/init.d/avahi-daemon stop
 * Stopping Avahi mDNS/DNS-SD Daemon avahi-daemon [ OK ]
.-(~)---------------------------------------------------------------------------------------------------------------------(hegemon@hextor)-
`--> sudo mt-daapd -f -c /etc/mt-daapd.conf
Firefly Version svn-1696: Starting with debuglevel 2
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-script.so: plugin declined to load
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: undefined symbol: avcodec_decode_audio
Plugin loaded: daap/svn-1696
Plugin loaded: rsp/svn-1696
Starting signal handler
Starting rendezvous daemon
Client connecting
Initializing database
Starting web server from /usr/share/mt-daapd/admin-root on port 3689
Registering rendezvous names
Could not create AvahiEntryGroup: Bad state
Could not create AvahiEntryGroup: Bad state
Could not create AvahiEntryGroup: Bad state
Serving 14501 songs. Startup complete in 1 seconds
Rescanning database
Could not create oggvorbis file handler
<...snip...>

[jcornwall (jay-jcornwall)]

Confirming this problem.

A standalone build of SVN 1737 (with HAVE_VA_COPY fix) works fine with Avahi.

[jcornwall (jay-jcornwall)]

OK, I had some time to debug this.

The problem is caused by a thread-unsafe patch introduced by the upstream maintainer (in debian/patches/13_avahi_fix_and_handle_restart.dpatch). This patch was intended to restore Firefly's broken Avahi handler so that daemon reconnections could be handled gracefully. By reenabling this broken part of Firefly, a slew of race conditions between Firefly and Avahi, which were already present in the code, were exposed.

The core of the problem is this: the avahi_simple_poll_* API makes it impossible to interact with from outside the event loop thread. This is by design and led to development of the thread-aware avahi_threaded_poll_* API.

I have put together a preliminary patch to resolve this (attached). It passes Valgrind's Helgrind race condition checker in all live paths I have tested; previously, crash-causing races were detected in several places. It has one weakness:

* _rend_avahi_lock/unlock are messily extended with a from_callback parameter. This was necessary because some locking code is shared by external threads and by the event thread (in callbacks) and Avahi provides no way to detect the thread context. Locking Avahi in the callback thread would otherwise lead to deadlock.

This is a criticism of elegance and not functionality. I have tested the patch and am fairly happy with it.

[Daniel Schierbeck (dasch)]

Confirmed on Jaunty with latest updates.

[elventear (elventear)]

I can confirm the problem.

A workaround is to first start mt-daapd and the avahi.

[mlaverdiere (mlaverdiere)]

Here's a "me too" statement.

The workaround suggested by elventear appears to work for me, although i,m experiencing various unusual cut-off when I'm using mt-daapd (Firefly media server).

[Andy Ruddock (andy-ruddock)]

Confirm that this bug affects AMD64, mt-daapd starts ok after avahi-daemon is stopped.

[mlaverdiere (mlaverdiere)]

I don't want to rush anything, but the workaround proposed (first start mt-daapd, then avahi) shows some limits for me (something seems to hang, once in a while) and I badly miss my home audio system (which is based on Firefly/Rocku Soundbridge) which used to work perfectly until Jaunty upgrade...

If anyone has an experimental patched package, I would gladly test it. Otherwise, I'm lost on trying to to compile SVN 1737 (I tried it, but it's not so obvious since it appears to rely on some autoconf magic) or on how to apply the proposed patch...

Thanks.

[jcornwall (jay-jcornwall)]

Ah, sorry. I dropped that patch in for the maintainer and omitted an explanation of how to apply it for users to experiment with.

You need to do this:

sudo apt-get install dpkg-dev fakeroot
sudo apt-get build-dep mt-daapd
apt-get source mt-daapd [no sudo on this one]
cd mt-daapd-0.9~r1696.dfsg
cp /path/to/downloaded/13_avahi_fix_and_handle_restart.dpatch debian/patches
dpkg-buildpackage -rfakeroot
sudo dpkg -i ../*.deb

After installing the new package you might find that 'apt-get upgrade' will want to reinstall the old version. You can either not install the new package and just copy the mt-daapd binary (bad but it'll work) or you can increment the package version number (bad because you might miss a real update). For the latter, do this before 'dpkg-buildpackage -rfakeroot':

sudo apt-get install devscripts
dch -i

[Sebastien Bacher (seb128)]

subscribing ubuntu-universe-sponsors so the patch gets reviewed, that's probably a jaunty sru candidate

[mlaverdiere (mlaverdiere)]

I think that this other bug report could be relevant, i.e. on Jaunty, avahi (and thus, Rhythmbox and other apps relying on avahi) doesn't show external shares : https://bugs.launchpad.net/ubuntu/+source/avahi/+bug/363768

[mlaverdiere (mlaverdiere)]

Forget about my previous comment. It turns out that the 363768 bug is related to a faulty ath9k driver, which has nothing to do with mt-daapd segfaulting with avahi.

[RecceDG (reccedg)]

Me too on the segfault.

I haven't tried the patch process yet - do you need more testers, or is this going to show up in an update?

DG

[RecceDG (reccedg)]

Here's the output:

sudo mt-daapd -f -c /etc/mt-daapd.conf
Firefly Version svn-1696: Starting with debuglevel 2
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: undefined symbol: avcodec_decode_audio
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-script.so: plugin declined to load
Plugin loaded: rsp/svn-1696
Plugin loaded: daap/svn-1696
Starting signal handler
Starting rendezvous daemon
Client running
Initializing database
Starting web server from /usr/share/mt-daapd/admin-root on port 3689
Registering rendezvous names
Segmentation fault

/var/log/messages:

kernel: [669257.257537] mt-daapd[3450]: segfault at 4 ip b74da6b6 sp bfbd5fc8 error 6 in libdbus-1.so.3.4.0[b74b5000+36000]

DG

[RecceDG (reccedg)]

Applying the patch per instructions got the server started.

DG

[mlaverdiere (mlaverdiere)]

Just in case it is helpful to someone, here's the package I have compiled (on Jaunty) with the patch provided by jcornwall (thanks a lot!). I've tested it and it works ...although I'm experiencing another problem, with avahi-daemon that I need to manually restart after each reboot. See this bug report: https://bugs.launchpad.net/ubuntu/+source/avahi/+bug/116984

[Dave Thiede (davidrthiede)]

i'm running jaunty release and the workaround of stopping the avahi-daemon doesn't seem to work on my configuration.

Linux quad 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64 GNU/Linux

I still get the messages about the plugins not loading.

2009-05-18 12:05:53 (d036c804): Error loading plugin /usr/lib/mt-daapd/plugins/ssc-script.so: plugin declined to load
2009-05-18 12:05:53 (d036c804): Error loading plugin /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: undefined symbol: avcodec_decode_audio

[jcornwall (jay-jcornwall)]

Those messages aren't a problem. (Well, they are, but they're unrelated to this bug!)

For completeness I've attached a patched AMD64 package. That should fix your problem.

[Dave Thiede (davidrthiede)]

jcornwall: I still haven't found a bug for my problem then. I tried your amd64 package but got the following:

dave@quad:~$ sudo aptitude install mt-daapd_0.9~r1696.dfsg-9ubuntu1_amd64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
E: Unknown pattern type: r
E: Unknown pattern type: r
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 0B of archives. After unpacking 0B will be used.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done

I then tried your instructions from comment 8 and couldn't get the build-dep working.

dave@quad:~$ sudo apt-get build-dep mt-daapd
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Build-dependencies for mt-daapd could not be satisfied.

I made a stab at manually adding the packages and ran into several dependency issues that were not reported normally but the apt tools. The build then proceeded but had multiple errors such as the following:

dpkg-shlibdeps: warning: symbol avcodec_decode_audio used by debian/mt-daapd/usr/lib/mt-daapd/plugins/ssc-ffmpeg.so found in none of the libraries.

should I create a new bug for my problem? from the initial poster documentation it is apparent that this same problem is present there also. as far as I can tell, if the ffmpeg plugin doesn't load then any files that require conversion (ogg at least) for itunes will fail. this also seems to affect display of mpeg movies to a tivo. it could be that the plugin failure is masking the real issue but I can't tell yet. the program worked great just prior to the upgrade.

[jcornwall (jay-jcornwall)]

Hi Dave,

Sorry, perhaps I was unclear about those warnings. They indicate a genuine bug in the mt-daapd package which has been around for quite a while in Ubuntu's package, from what I recall. It could indeed cause failures in transcoding if you rely on that feature and you should make a separate bug report if you need a fix for it.

This bug report is about the failure of mt-daapd to start without segfaulting in Jaunty. If you have no mt-daapd service at all then this report is relevant to you as well.

To install the package with a fix for that, try this:

sudo dpkg -i mt-daapd_0.9~r1696.dfsg-9ubuntu1_amd64.deb

(I haven't come across that build-dep error before. Perhaps it needs an extra repository enabled in Synaptic? The binary package should work for you, anyhow.)

[Dave Thiede (davidrthiede)]

fyi, I filed bug 378390 for the ffmpeg plugin load failure.

[Leon Barrett (leon-barrettnexus)]

This bug affects me too, but the patch fixes everything. I just thought I'd add my encouragement to the maintainers to apply the patch and update the official package. Thanks!

[Daniele Sdei (danielesdei)]

Hello,
I have installed the latest patched version from the official repositories but mt-daapd crashes on startup. This is my output:

$ sudo mt-daapd -f -c /etc/mt-daapd.conf
Firefly Version svn-1696: Starting with debuglevel 2
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: /usr/lib/mt-daapd/plugins/ssc-ffmpeg.so: undefined symbol: avcodec_decode_audio
Error loading plugin /usr/lib/mt-daapd/plugins/ssc-script.so: plugin declined to load
Plugin loaded: daap/svn-1696
Plugin loaded: rsp/svn-1696
Starting signal handler
Starting rendezvous daemon
Client running
Initializing database
Starting web server from /usr/share/mt-daapd/admin-root on port 3689
Registering rendezvous names
Segmentation fault

When I first stopped avahi-daemon then all function properly.

[Daniele Sdei (danielesdei)]

sorry, forgot to say that the ubuntu release is Jaunty Server 64 bit.
mt-daapd version is 0.9~r1696.dfsg-9
kernel version:
$ uname -rmo
2.6.28-11-server x86_64 GNU/Linux

[Ravindran K (ravindran-k)]

Hi all.

> mlaverdiere wrote on 2009-05-11: (permalink)
> * mt-daapd_0.9~r1696.dfsg-9_i386.deb (261.6 KiB, application/x-debian-package)

I was facing the same issue..

[13072.180379] mt-daapd[18382]: segfault at 4 ip b73b66b6 sp bfbb8028 error 6 in libdbus-1.so.3.4.0[b7391000+36000]
[13240.790376] mt-daapd[18767]: segfault at 4 ip b733b6b6 sp bfb3af68 error 6 in libdbus-1.so.3.4.0[b7316000+36000]
[13274.040396] mt-daapd[18783]: segfault at 4 ip b73db6b6 sp bffdd428 error 6 in libdbus-1.so.3.4.0[b73b6000+36000]

I installed the custom deb file (thanks mlaverdiere) and it works fine now.

cheers!!
Ravindran

[ian (ian6)]

I confess I don't know how maintaining universe packages in Ubuntu works, and I don't mean to sound ungrateful for your time and work, but is there anyone who can apply or veto this patch? Upgrading to Jaunty broke this package, and the bug has been open for 3.5 months with a patch fixing it available for 3.

Cheers.

[Toby Corkindale (tjc-wintrmute)]

Agreed.. Is there some step we have to take on the bug that we're missing?

[jcornwall (jay-jcornwall)]

I am not sure how the process works either.

One thing you could do is chase this upstream to the Debian maintainer who added the broken patch, which Ubuntu then inherited. If you look at my patch it is actually patching *another* patch which was added and maintained by Debian to fix a few problems.

It should be fixed there anyway (I had assumed a developer here would propagate this upstream to Debian) and if it is then it will probably be easier to get Ubuntu to adopt the patch.

[ian (ian6)]

Apparently this has already been fixed in Debian:

http://packages.debian.org/changelogs/pool/main/m/mt-daapd/mt-daapd_0.9~r1696.dfsg-14/changelog

   * debian/patches/13_avahi_fix_and_handle_restart.dpatch:
     + Rewritten; the previous fix made the code work on UP but uncovered
       the racy-by-design nature of the code. Switch to ThreadedPoll instead
       of SimplePoll.

So, Ubuntu? ;)

[Alvin (alvind)]

The version in Jaunty is 0.9~r1696.dfsg-9
The fixed version in Debian is 0.9~r1696.dfsg-11
The version in Karmic is 0.9~r1696.dfsg-13

So, I believe this is fixed.

[ian (ian6)]

Fixed in Karmic perhaps, but the package has been broken in Jaunty since... well, alpha 6 apparently.

[David McBride (david-mcbride)]

Indeed, still broken in Jaunty. A backport of the current Karmic package (which should be a trivial rebuild) would be most welcome.

[Michael Casadevall (mcasadevall)]

I marked the bug as fixed in karmic, and opened it on jaunty. Subscribing motu-sru for an ACK; I'll turn the existing patch into a debdiff and upload to proposed once I have an ACK.

[Benjamin Drung (bdrung)]

unsubscribing ubuntu-universe-sponsors for now

[John Dong (jdong)]

13_avahi_fix_and_handle_restart.dpatch looks reasonable (albeit a pretty hefty diff!), you have a preliminary ACK to translate that into a debdiff for SRU.

P.S. Please change the main bug description as required by the SRU procedure.

[Clint Byrum (clint-fewbar)]

Closing jaunty task as it is past EOL.