[trusty SRU/FFE] Add systemd binary package for snapd

[Martin Pitt]

Rationale: For backporting snapd to 14.04 LTS, we need to provide systemd's service manager (not just logind and auxiliary services like logind or timesyncd). upstart will continue to do the actual booting, and systemd will act as a "deputy init" which by default does not ship with/start any services by itself. We will only support this on server (at the first iteration at least), not on desktops.

Regression potential: This is a new binary package in universe, so existing systems are unaffected (provided that we ensure that the other binary packages do not change and there are no code changes that affect processes other than the "deputy pid 1" service manager). So for plain upgrades the regression potential is very low. However, there is a medium potential for breakage when actually installing the new systemd package, as it might interfere with upstart jobs or other running processes, cause boot/shutdown hangs, etc.

For init-system-helpers, the regression potential is near-zero: We never had (and never will) systemd as pid 1 in trusty, so deb-systemd-invoke was previously never called. It does get called now if you install Ubuntu packages that ship a systemd unit, so we need to make that a no-op to retain the current behaviour.

Test plan:
 1. Dist-upgrade a trusty installation to the proposed versions. Ensure this does not pull in "systemd", and that booting, shutdown, desktop startup, suspend on lid close, resume, logout, and user switching all still work.

 2. Install the "systemd" binary package (this will replace/remove systemd-shim). Verify that you can talk to the service manager with "sudo systemctl status". Check that booting and shutdown continues to work without (significant) delays.

 3. Ensure that "sudo journalctl" works and that "sudo systemctl status systemd-journald" is running and has a few lines of log at the end (unlike what you get when you run systemctl as user).

 4. Install a package that ships a systemd .service file, such as "haveged". Ensure that the service file is ignored, "pgrep -af haveged" should only have *one* process and "systemctl status haveged" should not be running (it should not exist, or not be enabled and be inactive). [This part also needs the updated init-system-helpers].

   The only services that are running are expected to be systemd-journald.service and systemd-journald.socket.

 5. Ensure that the standard targets are active, as that is where third-party/snap services hook into:

    systemctl status sysinit.target multi-user.target default.target

 6. Install snapd (not in trusty yet, e. g. from Thomas' PPA) and ensure you can install a snap, and its services start after installing the snap and after rebooting.

 7. Run "sudo apt-get install --reinstall systemd" to ensure that upgrades to newer systemd trusty versions work. The running systemd should *not* be restarted as that would disrupt snapd and its services (verify that the pid in "initctl status systemd" is the same before and after the upgrade).

 8. Run "sudo apt install -y colord && sudo apt purge -y colord". This should succeed.

 9. Dist-upgrade to 16.04 to ensure that there are no file conflicts, dependency issues, etc.

[Martin Pitt (pitti)]

The code for this is in https://git.launchpad.net/~ubuntu-core-dev/+git/systemd-trusty (I started with an import of the current trusty version). A corresponding package is in https://launchpad.net/~pitti/+archive/ubuntu/systemd .

This still does not match all the requirements from above: while it removes all *.services that systemd ships by itself, there are a lot of packages in trusty which ship their own, and these services are currently started by the deputy systemd (Examples: dbus.service, haveged.service, systemd-udevd.service) -- some of them fail on port conflicts, but some others cause a second process to run. This also does not start systemd right after package installation.

[Martin Pitt (pitti)]

> This also does not start systemd right after package installation.

Fixed in git.

Another issue: reboot hangs

[Martin Pitt (pitti)]

Status: Current git now removes/ignores /lib/systemd/system/ and only reads /lib/systemd/upstart/, where it ships the *.target's and journal services. It also adjusts the dbus.service shim and cleans this up.

Remaining known problem is that reboot is hanging. I haven't tested dist-upgrade to 16.04 yet.

[Martin Pitt (pitti)]

> Status: Current git now removes/ignores /lib/systemd/system/

Not entirely yet, linked units from /etc/systemd/system are still being read. This needs a bigger hammer.

[Martin Pitt (pitti)]

Status update:

> linked units from /etc/systemd/system are still being read. This needs a bigger hammer.

Fixed in git.

> reboot is hanging

This was due to /lib/lsb/init-functions.d/40-systemd which diverted SysV init scripts to systemctl which then ignored those as we want to disable SysV support for deputy systemd. Fixed in git.

> I haven't tested dist-upgrade to 16.04 yet.

Done now. It mostly works except for a two-minute hang in grub-common.postinst when trying to start grub-common.service. This gets into a deadlock due to the LSB hook reappearing while the deputy systemd is still running, and after the timeout grub-common is in a failed state. What works is to manually do "sudo initctl stop systemd; sudo rm -r /run/systemd/system; sudo apt-get -f install". After the dist-upgrade you need to run "sudo reboot -f".

I updated https://launchpad.net/~pitti/+archive/ubuntu/systemd to systemd 204-5ubuntu20.19pitti3 with the recent fixes, and this should now work reasonably well except for upgrades. Please test with snapd!

[Martin Pitt (pitti)]

git/PPA updated to 204-5ubuntu20.19pitti4 which now also fixes the hang during dist-upgrading to 16.04 LTS. "reboot -f" after the upgrade is still needed, though.

[Martin Pitt (pitti)]

Thomas discovered another problem:

# apt-get remove colord
Removing colord (1.0.6-1) ...
Failed to issue method call: Unit colord.service not loaded.
dpkg: error processing package colord (--remove):
 subprocess installed pre-removal script returned error exit status 5
Failed to issue method call: Unit colord.service failed to load: No such file or directory. See system logs and 'systemctl status colord.service' for details.
Errors were encountered while processing:
 colord
E: Sub-process /usr/bin/dpkg returned an error code (1)

The autogenerated prerm has:

if [ -d /run/systemd/system ]; then
 deb-systemd-invoke stop colord.service >/dev/null
fi

This previously was a no-op in trusty as systemd never ran as pid 1 and thus /run/systemd/system does not exist. But it does exist now with the deputy init, and:

  - We added a patch to ignore units shipped by packages (in /lib/systemd/system)

 - colord *only* ships a systemd unit, not a corresponding SysV script (as it's D-Bus activated), and thus this uses dh_systemd_start/deb-systemd-invoke instead of dh_installinit/invoke-rc.d.

As we don't ever expect deb-systemd-invoke to actually do something on trusty, I propose to just replace it with a /bin/true symlink. This is more robust and more efficient than trying to detect within it if systemd runs as deputy init.

[Martin Pitt (pitti)]

I uploaded that init-system-helpers fix to https://launchpad.net/~pitti/+archive/ubuntu/systemd . Once that gets uploaded to -proposed, I will add a versioned Breaks: to the new systemd binary.

[Martin Pitt (pitti)]

I discussed the current status with Thomas today, and he says that i-s-h and systemd are good to go. I uploaded those to the trusty-proposed SRU review queue.

[Steve Langasek (vorlon)]

Hello Martin, or anyone else affected,

Accepted init-system-helpers into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/init-system-helpers/1.14ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Martin, or anyone else affected,

Accepted systemd into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/204-5ubuntu20.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Po-Hsu Lin (cypressyew)]

Hello,

I tried to verify this with trusty server image on a laptop, everything act as expected before step 6 (except that stuff like desktop startup, suspend on lid close can't be tested with server image). After that it failed with:

ubuntu@ubuntu:~$ sudo snap install test-snapd-tools
error: cannot perform the following tasks:
- Setup snap "test-snapd-tools" (3) security profiles (cannot setup apparmor for snap "test-snapd-tools": cannot load apparmor profile "snap.test-snapd-tools.block": cannot load apparmor profile: exit status 1
apparmor_parser output:
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.test-snapd-tools.block in /var/lib/snapd/apparmor/profiles/snap.test-snapd-tools.block at line 305: syntax error, unexpected TOK_CONDLISTID, expecting TOK_MODE
)
- Setup snap "test-snapd-tools" (3) security profiles (cannot load apparmor profile "snap.test-snapd-tools.block": cannot load apparmor profile: exit status 1
apparmor_parser output:
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.test-snapd-tools.block in /var/lib/snapd/apparmor/profiles/snap.test-snapd-tools.block at line 305: syntax error, unexpected TOK_CONDLISTID, expecting TOK_MODE
)

snapd came from Thomas' ppa

[Martin Pitt (pitti)]

Thanks Po-Hsu. This is clearly a bug within test-snapd-tools, possibly fixed by bug 1641243. Is there a simpler "hello world" type snap that has a service that can be used to verify this without running into the apparmor profile issue?

Steps 7 to 9 can be done independently.

[Po-Hsu Lin (cypressyew)]

Hello Martin,

hello world does not work as well, but core snap can be installed without any problem.

ubuntu@ubuntu:~$ snap list
Name Version Rev Developer Notes
core 16.04.1 394 canonical -
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.13.0-102-generic #149-Ubuntu SMP Wed Nov 9 21:52:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ dpkg -l | grep systemd
ii libpam-systemd:amd64 204-5ubuntu20.20 amd64 system and service manager - PAM module
ii libsystemd-daemon0:amd64 204-5ubuntu20.20 amd64 systemd utility library
ii libsystemd-journal0:amd64 204-5ubuntu20.20 amd64 systemd journal utility library
ii libsystemd-login0:amd64 204-5ubuntu20.20 amd64 systemd login utility library
ii systemd 204-5ubuntu20.20 amd64 system and service manager
ii systemd-services 204-5ubuntu20.20 amd64 systemd runtime services
rc systemd-shim 6-2bzr1 amd64 shim for systemd

ubuntu@ubuntu:~$ sudo snap install hello-world
[sudo] password for ubuntu:
error: cannot perform the following tasks:
- Setup snap "hello-world" (27) security profiles (cannot setup apparmor for snap "hello-world": cannot load apparmor profile "snap.hello-world.env": cannot load apparmor profile: exit status 1
apparmor_parser output:
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.hello-world.env in /var/lib/snapd/apparmor/profiles/snap.hello-world.env at line 305: syntax error, unexpected TOK_CONDLISTID, expecting TOK_MODE
)
- Setup snap "hello-world" (27) security profiles (cannot load apparmor profile "snap.hello-world.env": cannot load apparmor profile: exit status 1
apparmor_parser output:
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.hello-world.env in /var/lib/snapd/apparmor/profiles/snap.hello-world.env at line 305: syntax error, unexpected TOK_CONDLISTID, expecting TOK_MODE
)

[Martin Pitt (pitti)]

snapd is not completely ready for trusty yet, as it needs quite a lot of adjustments to work in that old environment: https://github.com/snapcore/snapd/pull/2128 .

[Steve Langasek (vorlon)]

> ubuntu@ubuntu:~$ uname -a
> Linux ubuntu 3.13.0-102-generic #149-Ubuntu SMP Wed Nov 9 21:52:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

This shows that you are running the trusty release kernel. snapd will only be supported on trusty with the xenial hwe kernel. This probably explains your apparmor errors.

[Po-Hsu Lin (cypressyew)]

Oh OK!

As this bug was tagged with "verification-needed-trusty", I thought this one needs to be tested with Trusty 3.13 kernel.

Today I tried with 14.04.5 server image with the proposed 4.4.0-49-generic #70 hwe kernel. It works.

The apparmor error is a good hint, the apparmor package needs to be installed before installing the hello-world snap.

[Po-Hsu Lin (cypressyew)]

Forgot to mention that the apparmor, libapparmor-perl, libapparmor1 packages are from Thomas' ppa as well.

Verification step 7,8,9 are all good for the 4.4 hwe kernel on Trusty.

And I check with 3.13 kernel again, it works with apparmor package installed from Thomas' paa too.

As a result, I will set the tag to verification-done.
Thanks

[Martin Pitt (pitti)]

The verification of the Stable Release Update for init-system-helpers has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package init-system-helpers - 1.14ubuntu1

---------------
init-system-helpers (1.14ubuntu1) trusty; urgency=medium

  * script/deb-systemd-invoke: Replace with /bin/true. systemd as pid 1 and
    for /lib/systemd/system/ is not supported in Ubuntu 14.04. It will only
    be supported as "deputy init" running as an upstart job and handling
    /lib/systemd/upstart/ (and /{run,etc}/systemd/system as usual). This
    completely disables the handling of systemd units shipped by Ubuntu
    packages, to avoid suddenly breaking them when installing them alongside
    the new deputy systemd init. (LP: #1616422)
  * debian/init-system-helpers.manpages: Remove deb-systemd-invoke manpage as
    there is no POD any more to generate it from, and it's also irrelevant.

 -- Martin Pitt <email address hidden> Thu, 15 Sep 2016 13:04:42 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 204-5ubuntu20.20

---------------
systemd (204-5ubuntu20.20) trusty-proposed; urgency=medium

  * Build systemd binary package.
    Drop installation of /etc/* aside from systemd's own config files. This
    avoids a package conflict with systemd-services and we don't want to
    support the full feature set anyway. (LP: #1616422)
  * Disable SysV init support.
    This just gets in the way when running systemd as a "deputy init".
  * systemd: Add Conflicts: to systemd-shim
  * Create/use private D-Bus socket also for systemd --system.
    Without this we cannot use systemctl as root or when D-Bus is not running.
  * Do not read units from /lib/systemd/system, but from /lib/systemd/upstart/
    In Ubuntu 14.04 there are a lot of packages which ship a systemd system unit,
    but almost all of these must not run for running systemd's service manager as a
    "deputy" init alongside upstart. We do need some of them though, so read units
    from /lib/systemd/upstart.
    Only install the system units that we actually need for a deputy init (journal
    and all targets).
  * Add Breaks: to init-system-helpers that does not yet have a disabled
    deb-systemd-invoke, to complete the previous change.
  * Add upstart job for deputy systemd init.
    We also need to clean up /run/systemd/system after stop, so that things which
    check if systemd is running don't get confused.
  * Add dummy D-Bus units.
    These are built in for exposing systemd itself onto the system bus.
  * Drop LSB init hook.
    We must not redirect SysV init scripts to systemd when running as deputy init.
  * Stop systemd deputy upstart job on dist-upgrades.
    Also drop the removal guard as we do want to be able to remove the systemd
    package while it's only running the deputy init.
  * Update Vcs-Git: for new trusty git branch.

 -- Martin Pitt <email address hidden> Thu, 10 Nov 2016 15:14:54 +0100