Mumble server does not accept connections when started at machine boot time, works when service is restarted manually

[Benjamin Xiao]

I have an Ubuntu 14.04 desktop machine that has mumble-server installed. I have configured it to start at boot time using "sudo dpkg-reconfigure mumble-server". However, the server seems to not accept any connections, even though it is running. If I do a "sudo service mumble-server restart", then it starts accepting connections.

I have taken a look at the mumble logs and I've noticed that when mumble-server is started at boot time, it doesn't show a line saying that it is listening for connections.

=============================================================================
<W>2014-07-21 02:41:12.732 Initializing settings from /etc/mumble-server.ini (basepath /etc)
<W>2014-07-21 02:41:12.734 OpenSSL: OpenSSL 1.0.1f 6 Jan 2014
<C>2014-07-21 02:41:12.735 Successfully switched to uid 118
<W>2014-07-21 02:41:12.857 ServerDB: Opened SQLite database /var/lib/mumble-server/mumble-server.sqlite
<W>2014-07-21 02:41:12.859 Resource limits were 0 0
<W>2014-07-21 02:41:12.859 Successfully dropped capabilities
<W>2014-07-21 02:41:12.861 DBus registration succeeded
<W>2014-07-21 02:41:12.866 MurmurIce: Endpoint "tcp -h 127.0.0.1 -p 6502" running
<W>2014-07-21 02:41:12.932 Murmur 1.2.4 (1.2.4-0.2ubuntu1.1) running on X11: Ubuntu 14.04 LTS: Booting servers
<W>2014-07-21 02:41:12.964 1 => Announcing server via bonjour
<W>2014-07-21 02:41:12.994 1 => Not registering server as public
=============================================================================

However after I do a service restart, I see a "Server listening on [::]:64738" line in the logs. The server is able to accept connections when this line is present.

=============================================================================
<W>2014-07-21 02:44:49.614 MurmurIce: Shutdown complete
<W>2014-07-21 02:44:49.667 Initializing settings from /etc/mumble-server.ini (basepath /etc)
<W>2014-07-21 02:44:49.668 OpenSSL: OpenSSL 1.0.1f 6 Jan 2014
<C>2014-07-21 02:44:49.668 Successfully switched to uid 118
<W>2014-07-21 02:44:49.705 ServerDB: Opened SQLite database /var/lib/mumble-server/mumble-server.sqlite
<W>2014-07-21 02:44:49.706 Resource limits were 0 0
<W>2014-07-21 02:44:49.707 Successfully dropped capabilities
<W>2014-07-21 02:44:49.708 DBus registration succeeded
<W>2014-07-21 02:44:49.710 MurmurIce: Endpoint "tcp -h 127.0.0.1 -p 6502" running
<W>2014-07-21 02:44:49.750 Murmur 1.2.4 (1.2.4-0.2ubuntu1.1) running on X11: Ubuntu 14.04 LTS: Booting servers
<W>2014-07-21 02:44:49.783 1 => Server listening on [::]:64738
<W>2014-07-21 02:44:49.810 1 => Announcing server via bonjour
<W>2014-07-21 02:44:49.844 1 => Not registering server as public
=============================================================================

This bug is really annoying because I have to make sure to do a service restart every time I reboot the machine. This machine is meant as a personal server and it needs to be up constantly. Sometimes I'll forget to do a mumble-server restart and mumble is basically unavailable to everyone who tries to connect.

My theory is that because I am using Network Manager to connect to the internet, maybe my interface isn't ready by the time mumble-server starts? I am using an SSD drive, so maybe mumble-server is starting up too fast and there is a race condition. Regardless, mumble-server should automatically start listening for connections when it detects that the network is up.

Steps to reproduce:
1.) Install mumble-server
2.) Do a "sudo dpkg-reconfigure mumble-server" and set it to start on boot with high priority privileges
3.) Restart the machine
4.) Try connecting to the server using a mumble client. This will not work.
5.) Do a "sudo service mumble-server restart".
6.) Connect again and verify that it works.

Expected behavior:
1.) Mumble server should accept connections when the service is started at boot time.

Current behavior:
1.) Mumble server does not accept connections when it is started at boot time. It starts working only after a manual service restart.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: mumble-server 1.2.4-0.2ubuntu1.1
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
Date: Mon Jul 21 10:41:59 2014
InstallationDate: Installed on 2014-06-11 (40 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
ProcEnviron:
 LANGUAGE=en_US
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: mumble
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.default.mumble.server: [modified]
mtime.conffile..etc.default.mumble.server: 2014-07-21T02:40:32.651299

[Benjamin Xiao (ben-r-xiao)]

I have managed to workaround the issue by creating a NetworkManager dispatcher.d script. It basically restarts the mumble-server when it detects that my connection is up.

Put this script into /etc/NetworkManager/dispatcher.d/90mumblerestart
=============================================================

#!/bin/sh -e

IF=$1
STATUS=$2

if [ "$IF" = "em1" ]
then
        case "$STATUS" in
                up)
                        service mumble-server restart
                ;;
                *)
                ;;
        esac
fi

====================================================

Of course, replace em1 with whatever network interface you're using. This script works around the problem for me and I am able to connect to the mumble-server whenever I restart. The only downside is that whenever you disconnect/reconnect on your connection, the mumble server will get restarted, but I figured that if you're doing that all clients will be dropped anyways so it really makes no difference.

This is a workaround, but it is by no means a correct way to fix this bug.

[Fred Barnes (fb5014)]

i am running ubuntu 64bit 14.04 my mumble-server worked untill i installed mumble-django, i fixed mumbled-jango but my mumble-server still won't let anyone connect, it says connection timed out, i have tried uninstalling and purging and trying the 1.3.0, even tried building from their repo on bitbucket, nothing works. mumble server runs but nothing is able to connect..

[Trey (tdpeek3)]

I am running ubuntu 64bit 14.04 and my mumble-server is behaving the same way as the OP described. My only work around has been to not modify the ini configuration file (leave everything with the default settings). I've tested this with many reboots and as long as I don't modify the /etc/mumble-server.ini file it will reconnect during reboot. The moment this file is modified I must run service mumble-server restart to get it to the mumble server to start accepting connections. Like the OP I also noticed that my log files were missing when the server wouldn't start automatically.

1 => Server listening on [::]:64738

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Knadle (chris-knadle)]

I don't know how I got to be the maintainer of this package in Ubuntu; I assume it's becaue I'm the maintainer for the pacakge in Debian.

I've installed Ubuntu 14.04 LTS Desktop edition into a VirtualBox VM and am not able to reproduce this bug, regardless of whether I modify the mumble-server.ini file or not; mumble-server restarts upon machine restart in either case.

Re: Steps to reproduce:
    [...]
    2.) Do a "sudo dpkg-reconfigure mumble-server" and set it to start on boot with high priority privileges

What is the intended purpose for this change? Does this change the bootup order?

[Benjamin Xiao (ben-r-xiao)]

Step 2 I think was to get it to start on boot. If it's automatically loaded on boot now then I guess you can ignore that step.

I think the main issue is that there's a race condition where mumble-server starts up before the network is ready. It may very well work in VirtualBox due to when the network gets initialized, but it is probably different on actual hardware. Restarting the server after boot always works.

In the logs, do you see a "Server listening on [::]:64738" message? If you do, then you're not hitting the bug. The problem only occurs when that line is missing from the logs.

[Chris Knadle (chris-knadle)]

Yes, it started on boot (and reboot) by default.

And yes, I'm getting "Server listening on [::]:64738" everytime, so I'm not hitting the bug. :-/

[Benjamin Xiao (ben-r-xiao)]

Can you try disabling your connection first, start mumble, and then restart your connection?

My theory is that when mumble is started without a valid connection that's up, it won't show that log message and thus will fail to connect any clients.

[Chris Knadle (chris-knadle)]

Yep, that duplicated the bug; mumble-server will start, does not show "Server Listening", and indeed 'netstat -tpln' shows it's listening only locally and on a different port:
tcp 0 0 127.0.0.1:6502 0.0.0.0:* LISTEN 2360/murmurd

That's the Ice port that is shown in the log:
<W>2015-03-20 22:32:28.962 MurmurIce: Endpoint "tcp -h 127.0.0.1 -p 6502" running

On the Ubuntu 14.04 mumble-server package, Ice is enabled by default and set to listen on localhost only [on the current mumble-server package in Debian Sid and Jessie (1.2.8-2) that is about to be released, Ice is disabled by default]. So by default without networking available, murmurd starts but is only listening on the Ice port. If Ice is disabled in /etc/mumble-server.ini in Ubuntu then the init script will still run but murmurd will *not* be started at all, and the log doesn't seem to indicate that.

I think what's going on is that there's only an init script for mumble-server right now -- there haven't been any changes made for upstart nor systemd. The init script indicates that networking is required before starting:
   # Required-Start: $network $local_fs $remote_fs
but it seems that upstart and systemd may not be parsing these comments to delay the startup of mumble-server (murmurd) until networking is available.

So to fix this I think requires patching mumble-server to add upstart and systemd service files so that those init systems are aware that they need to delay the startup of mumble-server until networking is available. I'm likely going to focus on systemd first because that's the default in Debian (and now Ubuntu) but in the meantime if someone has an upstart service file for mumble-server I'd be glad to see if I can figure out how to get it into Ubuntu 14.04 at some point. [At preasent I'm not a Debian Developer nor a Debian Maintainer -- I'm "just a guy" -- so I need a DD to sponsor an upload even for Debian. ;-)]

[Chris Knadle (chris-knadle)]

I think I've found a workaround, though it defies logic. In /etc/mumble-server.ini set:

   host=::

which will work for both IPv6 and IPv4 -- or to limit use to only IPv4, use a setting of:

   host=0.0.0.0

[This works on Debian with mumble-1.2.8-2 at least.]

The reason this makes no sense is that this is exactly what murmurd tries to do in

src/murmur/Meta.cpp:234:
---------------------------------------------------------------------
      if (hasipv6) {
         qlBind << QHostAddress(QHostAddress::AnyIPv6);
#ifdef Q_OS_UNIX
         if (hasipv4) {
            int s = ::socket(AF_INET6, SOCK_STREAM, 0);
                if (s != -1) {
                   int ipv6only = 0;
                   socklen_t optlen = sizeof(ipv6only);
                   if (getsockopt(s, IPPROTO_IPV6, IPV6_V6ONLY,
&ipv6only, &optlen) == 0) {
                      if (ipv6only == 0)
                          hasipv4 = false;
                   }
                   close(s);
               }
         }
#endif
      }
      if (hasipv4)
         qlBind << QHostAddress(QHostAddress::Any);
      }
---------------------------------------------------------------------

The "Any" and "AnyIPv6" addresses are supposed to mean "0.0.0.0" and
"::" like one would expect:

   http://doc.qt.io/qt-4.8/qhostaddress.html#SpecialAddress-enum

I went through the effort of creating a systemd service file with a Wants=network-online.target and After=network-online.target and unfortunately that doesn't seem to help this in the least, so it's likely that an upstart service file wouldn't help either.

For reference, the Debian bug is #780300:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=780300

[J J (goodguyjpspam)]

I am experiencing this problem on my Raspberry Pi 2. By changing permissions to the /var/run/mumble-server to 777, it works.

Maybe this can be investigated as I lack the skills to go any further.

[Chris Knadle (chris-knadle)]

On 06/28/2015 04:41 PM, J J wrote:
> I am experiencing this problem on my Raspberry Pi 2. By changing
> permissions to the /var/run/mumble-server to 777, it works.
>
> Maybe this can be investigated as I lack the skills to go any further.

When mumble-server (murmurd) starts it needs to write its process ID to
/var/run/mumble-server/mumble-server.pid so that the init script can use
this file to check if the server is running, or to know which PID to
stop to shut mumble-server down. This shouldn't have any effect on
whether the mumble-server daemon accepts connections, so this sounds
unrelated to this particular bug.

   -- Chris

--
Chris Knadle
<email address hidden>