Fresh install of livepatch doesn't recognize ubuntu xenial

Bug #1667470 reported by Andreas Hasenack
72
This bug affects 15 people
Affects Status Importance Assigned to Milestone
Canonical Livepatch Client
Fix Released
Critical
Chad Smith
snap-confine
Fix Released
Undecided
Zygmunt Krynicki

Bug Description

A fresh install of livepatch on an updated xenial ubuntu machine doesn't recognize it and refuses to start:
Feb 23 20:01:09 node-6 /usr/bin/snap[25852]: cmd.go:111: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
Feb 23 20:01:09 node-6 snap[25852]: 2017/02/23 20:01:09 Only Ubuntu 16.04 LTS is supported, exiting.

Some debugging showed that /usr/bin/lsb_release doesn't exist in the latest core snap (see https://bugs.launchpad.net/snappy/+bug/1619420 and comments #11 and #14). livepatch uses that to determine if it can be used or not. Without it, it fails and shows that error message about not running on xenial.

Existing systems that upgraded to that core snap are not affected unless they reboot. If they reboot, they can "snap revert core" and reboot one more time to restore livepatch functionality by using the old core snap. Probably not a very useful workaround since the goal of livepatch is to prevent reboots, but hey :)

Tags: landscape
Revision history for this message
Chad Smith (chad.smith) wrote :

Confirmed that we can't use /etc/os-release directly from a snap as it posts snap-related information only. We instead need to parse /etc/lsb-release

dash-shell.dash
$ cat /etc/os-release
NAME="Ubuntu Core"
VERSION="16"
ID=ubuntu-core
PRETTY_NAME="Ubuntu Core 16"
VERSION_ID="16"
HOME_URL="http://www.snapcraft.io/"
BUG_REPORT_URL="http://bugs.launchpad.net/snappy/"

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

Changed in canonical-livepatch-client:
importance: Undecided → Critical
status: New → Confirmed
description: updated
Revision history for this message
Francis Ginther (fginther) wrote :

Confirmed failure can be observed with the livepatch-client system_test. I did have to tweak system-tests to get them working:

$ git diff
diff --git a/tests/system_test.go b/tests/system_test.go
index 72c8d36..7f34364 100644
--- a/tests/system_test.go
+++ b/tests/system_test.go
@@ -529,7 +529,7 @@ func init() {
        // Create test system
        Vm = SystemTest{Name: "livepatchd-vm", User: "ubuntu", Series: "xenial",
                Arch: "amd64", Flavor: "generic", PackageVersion: TestPackageVersion,
- KernelVersion: "4.4.0-21.37", KernelABI: "4.4.0-21"}
+ KernelVersion: "4.4.0-63.84", KernelABI: "4.4.0-63"}
        Vm.setup()
 }

The 4.4.0-21.37 kernel was failing to boot in the KVM.

The error in /var/log/syslog:
Feb 23 20:49:01 livepatchd-vm snap[1939]: 2017/02/23 20:49:01 Only Ubuntu 16.04 LTS is supported, exiting.
Feb 23 20:49:01 livepatchd-vm systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Main process exited, code=exited, status=1/FAILURE
Feb 23 20:49:01 livepatchd-vm systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Unit entered failed state.
Feb 23 20:49:01 livepatchd-vm systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Failed with result 'exit-code'.
Feb 23 20:49:01 livepatchd-vm systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Service hold-off time over, scheduling restart.
Feb 23 20:49:01 livepatchd-vm systemd[1]: Stopped Service for snap application canonical-livepatch.canonical-livepatchd.
Feb 23 20:49:01 livepatchd-vm systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Start request repeated too quickly.
Feb 23 20:49:01 livepatchd-vm systemd[1]: Failed to start Service for snap application canonical-livepatch.canonical-livepatchd.

description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Reason why upgraded systems kept working: https://bugs.launchpad.net/snapd/+bug/1667479

description: updated
description: updated
Chad Smith (chad.smith)
Changed in canonical-livepatch-client:
assignee: nobody → Chad Smith (chad.smith)
David Britton (dpb)
Changed in canonical-livepatch-client:
status: Confirmed → In Progress
information type: Proprietary → Public
Revision history for this message
John Lenton (chipaca) wrote :

> Confirmed that we can't use /etc/os-release directly from a snap as it posts
> snap-related information only. We instead need to parse /etc/lsb-release

Chad, I'm not sure I follow what you mean with this. Could you explain?

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

We should fix this in snap-confine by bind mounting distro's /etc/os-release (after resolving the symlink) on top of /etc/os-release in the core snap.

/home/zyga # mount --bind /var/lib/snapd/hostfs/usr/lib/os-release /etc/os-release
/home/zyga # cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Changed in snap-confine:
assignee: nobody → Zygmunt Krynicki (zyga)
status: New → Confirmed
Revision history for this message
John Lenton (chipaca) wrote :

(I chatted about this with Zyga, understand Chad's point now)

As a workaround the snap can use the os-release from hostfs directly.

Revision history for this message
John Lenton (chipaca) wrote :

(except your snap can't read that file? drat)

Revision history for this message
Chad Smith (chad.smith) wrote :

Fix just landed, going through publishing the snap now https://github.com/CanonicalLtd/livepatch-client/pull/8

Revision history for this message
Chad Smith (chad.smith) wrote :

Publishing initially to --beta channel for testing.

Revision history for this message
Chad Smith (chad.smith) wrote :

Will rollout the fix to --release just after we confirm an automated snap refresh fixes this without additional interaction on the client

Chad Smith (chad.smith)
Changed in canonical-livepatch-client:
status: In Progress → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote :

canonical-livepatch snap version 7 revision 21 Fixes this and relies on /etc/lsb-release instead of the lsb_release tool.

Old kernels <= 4.4.0-24 might get hit with a snap mount kernel oops, a reboot of the system resolved this on our end.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1572316

Changed in canonical-livepatch-client:
status: Fix Committed → Fix Released
Revision history for this message
Oliver Grawert (ogra) wrote :

long term please use /etc/os-release.
since we do not ship lsb_release at all in the core snap it is not guaranteed that /etc/lsb-release will stay around ... (i will not remove it without notification though, but please plan to move over once snap-confine will provide a bind mount of the hosts os-release file on classic installs)

Revision history for this message
Chad Smith (chad.smith) wrote :

ogra, we are using /etc/lsb-release on the base system, not looking specifically in the core snap. So, I'm not sure comment #12 applies does it?

Revision history for this message
Chad Smith (chad.smith) wrote :

We had a discussion about using the core snaps /etc/lsb-release and decided we didn't want to get hit by this again because, long term, the snap should only be reporting about snap-related configuration and environment. We can rely on etc/lsb-release on the host system right?

Revision history for this message
Chad Smith (chad.smith) wrote :

Ahh, we will watch for the fix related to binding /etc/os-release from the underlying system in comment #5. When that lands we will change livepatch client to parse this file in the snap.

Revision history for this message
Oliver Grawert (ogra) wrote :

i dont think your snap can actually see the hosts lsb-release unless it is either "confinement: classic" or snap-confine bind-mounts the right file for you on top of the one in the core snap.

Revision history for this message
Chad Smith (chad.smith) wrote :

Hmm, our snap is confinement:strict with the following plugs.

        plugs: [ network-bind, network-control, kernel-module-control, system-observe, hardware-observe ]

We are able to manipulate /etc/lsb-release on the host system to cause the snap to claim it doesn't support the underlying system, so I believe hardware-observe is probably mounting that for us.

Revision history for this message
John Lenton (chipaca) wrote :

In a classic system, when running a confinement:strict snap, most of your /etc/ inside the snap is the /etc from the classic system. This includes both /etc/lsb-release and /etc/os-release.

However, /etc/os-release is a symlink to ../usr/lib/os-release, which when inside the snap is the core snap's. We'll be explicitly mounting the "outside" os-release over that symlink to make this kind of thing work.

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

This will be fixed shortly :-)

Changed in snap-confine:
status: Confirmed → In Progress
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

This is being fixed in the following PR: https://github.com/snapcore/snapd/pull/2947

Changed in snap-confine:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.