MAAS does not detect properly if Ubuntu is using upstart/systemd

Bug #1732703 reported by Victor Tapia on 2017-11-16
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
1.9
Critical
Andres Rodriguez
maas (Ubuntu)
Critical
Unassigned

Bug Description

[impact]
Since Trusty uses upstart by default, MAAS manages its services with upstart. However, when a user installs systemd (even if it is not used as the init system), MAAS detects systemd installed and tries to manage its services via systemd. This obviously creates issues and prevents MAAS from working.

[Test Case]
1. Install & configure MAAS
2. Add machines
3. install systemd
4. MAAS will fail to manage machines

[Regression potential]
Minimal. This just ensures that upstart is detected correctly even if systemd is installed (but not used).

[Original bug report]
Trusty uses upstart by default, and installing snapd (e.g. for livepatch purposes), pulls systemd too. In this setup, upstart is _not_ replaced by systemd, but MAAS "detects" systemd as init because of the existence of /run/systemd/system:

@src/provisioningserver/utils/__init__.py:505

SYSTEMD_RUN_PATH = '/run/systemd/system'

def get_init_system():
    """Returns 'upstart' or 'systemd'."""
    if os.path.exists(SYSTEMD_RUN_PATH):
        return 'systemd'
    else:
        return 'upstart'

One possible solution would be to check if /sbin/init is a symlink pointing to /lib/systemd/systemd:

def get_init_system():
    """Returns 'upstart' or 'systemd'."""
    initpath = os.readlink("/sbin/init")
    if (initpath == "/lib/systemd/systemd"):
        return 'systemd'
    else:
    return 'upstart'

Other affected parts of the code are the postinst files for maas-proxy and maas-dhcp (debian/maas-proxy.postinst debian/maas-dhcp.postinst), throwing an error if maas is installed after systemd in Trusty

Tags: sts Edit Tag help

Related branches

Changed in maas:
status: New → Incomplete
status: Incomplete → Won't Fix
Andres Rodriguez (andreserl) wrote :

Hi Victor,

Can you clarify what issues you are having? It is not clear from the bug the issues you are having.

That said, MAAS was never enabled to work on a system with both systemd + upstart, where upstart would remain the init system. Ubuntu, at the time, was not even fully prepared to use systemd. This is a grey area where it is difficult to fix something that is not supported, with a high regression potential.

Victor Tapia (vtapia) wrote :

Hi Andres,

At the moment, I can see the service monitor complain:

Nov 16 12:36:48 trusty-maas maas.dhcp: [ERROR] DHCPv4 server failed to restart (for network interfaces eth0): Unable to parse the output from systemd for service 'maas-dhcpd'.
Nov 16 12:36:52 trusty-maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to parse the output from systemd for service 'tgt'.
Nov 16 12:38:38 trusty-maas maas.service_monitor: [ERROR] While monitoring service 'maas-dhcpd' an error was encountered: Unable to parse the output from systemd for service 'maas-dhcpd'.
Nov 16 12:38:38 trusty-maas maas.service_monitor: [ERROR] While monitoring service 'maas-dhcpd6' an error was encountered: Unable to parse the output from systemd for service 'maas-dhcpd6'.
Nov 16 12:38:38 trusty-maas maas.service_monitor: [ERROR] While monitoring service 'tgt' an error was encountered: Unable to parse the output from systemd for service 'tgt'.

I'm currently building a reproducer to see if commissioning/deploying is somehow affected.

Victor Tapia (vtapia) wrote :

Trying to enlist a machine when systemd+upstart are installed in trusty, throws this error:

==> /var/log/maas/clusterd.log <==
2017-11-21 11:42:45+0000 [-] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the RemoveHostMaps command. [trusty-maas:pid=844:cmd=RemoveHostMaps:ask=25]
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 200, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/dhcp.py", line 237, in remove_host_maps
            if not _is_dhcpv4_managed_and_active(CannotRemoveHostMap):
          File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/dhcp.py", line 164, in _is_dhcpv4_managed_and_active
            if service_monitor.get_service_state("dhcp4") != SERVICE_STATE.ON:
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 200, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/service_monitor.py", line 120, in get_service_state
            return self._get_service_status(service)[0]
          File "/usr/lib/python2.7/dist-packages/provisioningserver/service_monitor.py", line 216, in _get_service_status
            service.service_name)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/service_monitor.py", line 265, in _get_systemd_service_status
            service_name))
        provisioningserver.service_monitor.ServiceParsingError: Unable to parse the output from systemd for service 'maas-dhcpd'.

==> /var/log/maas/regiond.log <==
2017-11-21 11:42:45 [-] Error on request (63) node.create: ('UNHANDLED', 'Unknown Error [trusty-maas:pid=844:cmd=RemoveHostMaps:ask=25]')
        Traceback (most recent call last):
        Failure: twisted.protocols.amp.UnhandledCommand: ('UNHANDLED', 'Unknown Error [trusty-maas:pid=844:cmd=RemoveHostMaps:ask=25]')

A similar error is shown when trying to commission (same ServiceParsingError for maas-dhcpd). Removing systemd, or patching get_init_system() to detect the proper service file, makes the issue disappear.

Victor Tapia (vtapia) wrote :

Commissioning logs: https://pastebin.canonical.com/203632/

The previous patch gave an error when only upstart was running, this should cover that case too (thanks ivanhitos):

def get_init_system():
    """Returns 'upstart' or 'systemd'."""
    if os.path.islink("/sbin/init"):
        initpath = os.readlink("/sbin/init")
        if initpath == "/lib/systemd/systemd":
            return 'systemd'
        else:
            return 'upstart'
    else:
        return 'upstart'

Drew Freiberger (afreiberger) wrote :

I can confirm that patch in #4 works on MAAS 1.9.5 on trusty.

description: updated
Changed in maas (Ubuntu):
importance: Undecided → Critical
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers