pro attaches but fails to enable services

[Mariusz Matuszek]

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.7 LTS"

# pro attach [REDACTED]
Enabling default service esm-apps
Enabling default service esm-infra
Enabling default service livepatch
Stderr: error executing config: livepatchd error: snappy kernel-module-control interface not connected

Stdout:
Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: error executing config: livepatchd error: snappy kernel-module-control interface not connected

ERROR: Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: error executing config: livepatchd error: snappy kernel-module-control interface not connected

Failed to enable default services, check: sudo pro status

# pro status
SERVICE ENTITLED STATUS DESCRIPTION
cc-eal yes disabled Common Criteria EAL2 Provisioning Packages
cis yes disabled Security compliance and audit tools
esm-apps yes disabled Expanded Security Maintenance for Applications
esm-infra yes disabled Expanded Security Maintenance for Infrastructure
fips yes disabled NIST-certified core packages
fips-updates yes disabled NIST-certified core packages with priority security updates
livepatch yes disabled Canonical Livepatch service
ros yes disabled Security Updates for the Robot Operating System
ros-updates yes disabled All Updates for the Robot Operating System

For a list of all Ubuntu Pro services, run 'pro status --all'
Enable services with: pro enable <service>

     Account: [REDACTED]
Subscription: Ubuntu Pro - free personal subscription

# pro enable esm-apps
One moment, checking your subscription first
Unexpected APT error. See /var/log/ubuntu-advantage.log

Excerpt from log (perhaps related):

"DEBUG", "root", "subp", 640, "Failed running command '/usr/lib/apt/apt-helper download-file https://bearer:<REDACTED>@esm.ubuntu.com/apps/ubuntu/pool/ /tmp/tmpqlzncn5d/apt-helper-output' [exit(100)]. Message: W: Can't drop privileges for downloading as file '/tmp/tmpqlzncn5d/apt-helper-output' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)\nE: Failed to fetch https://esm.ubuntu.com/apps/ubuntu/pool/ The HTTP server sent an invalid reply header\n\nE: Download Failed\n Retrying 3 more times.", {}]

# ls -la /etc/apt/auth.conf.d/
total 8
drwxr-xr-x 2 root root 4096 maj 21 2019 .
drwxr-xr-x 7 root root 4096 sie 9 12:20 ..

# cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ xenial-backports universe main multiverse restricted
deb http://archive.ubuntu.com/ubuntu/ xenial multiverse restricted main universe
deb http://archive.ubuntu.com/ubuntu/ xenial-updates multiverse main restricted universe
deb http://security.ubuntu.com/ubuntu/ xenial-security main restricted multiverse universe
# deb-src http://archive.ubuntu.com/ubuntu/ xenial-backports main universe multiverse restricted
# deb-src http://archive.ubuntu.com/ubuntu/ xenial main universe multiverse restricted
# deb-src http://archive.ubuntu.com/ubuntu/ xenial-updates main universe multiverse restricted
# deb-src http://security.ubuntu.com/ubuntu/ xenial-security main universe multiverse restricted

# apt update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [99,8 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Fetched 99,8 kB in 0s (145 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

# apt-cache policy ubuntu-advantage-tools
ubuntu-advantage-tools:
  Installed: 28.1~16.04
  Candidate: 28.1~16.04
  Version table:
 *** 28.1~16.04 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status

# apt-cache policy ubuntu-advantage-pro
ubuntu-advantage-pro:
  Installed: 28.1~16.04
  Candidate: 28.1~16.04
  Version table:
 *** 28.1~16.04 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
        100 /var/lib/dpkg/status
     27.5~16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages

[Mariusz Matuszek (mariuszxc)]

If I can provide any further relevant info just let me know what you need.

[Grant Orndorff (orndorffgrant)]

Hello Mariusz and thank you for reporting this issue!

I'm unable to reproduce this issue so it may be something about your environment. To help us figure out what is going on, can you run the following command:

/usr/lib/apt/apt-helper download-file https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release /tmp/test

If that also fails, then that tells us the problem is likely in apt/apt-helper rather than in `pro` or esm.ubuntu.com

Also, do you have any proxies configured via environment variables or /etc/apt/apt.conf.d/ or via `pro config set ...` or otherwise? Is there a MITM firewall in your environment?

For the canonical-livepatch error "error executing config: livepatchd error: snappy kernel-module-control interface not connected", I'll contact the livepatch team to get their opinion.

[Mariusz Matuszek (mariuszxc)]

Hello Grant and thanks for fast reaction. No MITM (that I know) of and no proxies, but it seems your intuition was perfect:

$ /usr/lib/apt/apt-helper download-file https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release /tmp/test
Err:1 https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release
  The HTTP server sent an invalid reply header
E: Failed to fetch https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release The HTTP server sent an invalid reply header

E: Download Failed
$ ls -l /tmp/test
-rw-r--r-- 1 xxx xxx 0 sie 9 15:50 test

So I did this test:

$ wget https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release
--2023-08-09 15:51:54-- https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release
Resolving esm.ubuntu.com (esm.ubuntu.com)... 91.189.91.47, 185.125.190.75, 91.189.91.46, ...
Connecting to esm.ubuntu.com (esm.ubuntu.com)|91.189.91.47|:443... connected.
ERROR: cannot verify esm.ubuntu.com's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Unable to locally verify the issuer's authority.
To connect to esm.ubuntu.com insecurely, use `--no-check-certificate'.

and finally:

$ wget --no-check-certificate https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release
--2023-08-09 15:55:33-- https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release
Resolving esm.ubuntu.com (esm.ubuntu.com)... 185.125.190.23, 91.189.91.47, 91.189.91.46, ...
Connecting to esm.ubuntu.com (esm.ubuntu.com)|185.125.190.23|:443... connected.
WARNING: cannot verify esm.ubuntu.com's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 6639 (6,5K) [application/octet-stream]
Saving to: ‘Release’

Release 100%[===========================================================================================>] 6,48K --.-KB/s in 0s

2023-08-09 15:55:33 (2,37 GB/s) - ‘Release’ saved [6639/6639]

So it seems I am missing some CA related updates.
What can I do about it?

[Grant Orndorff (orndorffgrant)]

Thank you for figuring that out!

So it looks like your system doesn't trust Let's Encrypt, whose root CA is ISRG Root X1 (https://letsencrypt.org/certificates/)

You can test that your system does/doesn't trust that root CA more directly with their test domain:

wget https://valid-isrgrootx1.letsencrypt.org/

I was able to get a xenial system into a state where it didn't trust the ISRG Root X1 by downgrading and upgrading the ca-certificates package.

If you would like your system to trust the ISRG Root X1 CA, then try the following:

First, make sure ca-certificates is up to date

sudo apt update && sudo apt install ca-certificates

If there was an update, that may have fixed it, test with wget https://valid-isrgrootx1.letsencrypt.org/

If that still fails, you can run

sudo dpkg-reconfigure ca-certificates

You can select "ask" and then scroll down to the ISRG Root X1 cert and select it using the spacebar before hitting enter. That will add ISRG Root X1 to your list of trusted root certificates.

Now try again: wget https://valid-isrgrootx1.letsencrypt.org/

If that works, then please try `sudo pro detach && sudo pro attach $token` and let me know if anything is still failing. I'm not sure if this will fix the livepatch error.

[Mariusz Matuszek (mariuszxc)]

So wget test failed:

wget https://valid-isrgrootx1.letsencrypt.org/
--2023-08-09 18:15:47-- https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org (valid-isrgrootx1.letsencrypt.org)... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org (valid-isrgrootx1.letsencrypt.org)|52.9.173.94|:443... connected.
ERROR: cannot verify valid-isrgrootx1.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Unable to locally verify the issuer's authority.

which further confirmed your evaluation. Performed 'apt update', then

# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
#
# You can verify the status of security fixes using the `pro fix` command.
# E.g., a recent Ruby vulnerability can be checked with: `pro fix USN-6219-1`
# For more detail see: https://ubuntu.com/security/notices/USN-6219-1
#
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

So it appears the package is up to date, but with 'aptitude' I verified, that ca-certificates was _not_ in 'held' state (it wasn't, just standard 'i' flag), then:

# apt-cache policy ca-certificates
ca-certificates:
  Installed: 20211016ubuntu0.16.04.1sav0
  Candidate: 20211016ubuntu0.16.04.1sav0
  Version table:
 *** 20211016ubuntu0.16.04.1sav0 100
        100 /var/lib/dpkg/status
     20210119~16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages
     20160104ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages

confirms I have the latest available (known to apt) installed.

(I know it is a different beast, but also verified the ca-certificates-java for completeness:
# apt-cache policy ca-certificates-java
ca-certificates-java:
  Installed: 20160321ubuntu1
  Candidate: 20160321ubuntu1
  Version table:
 *** 20160321ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
        100 /var/lib/dpkg/status
     20160321 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages
)

Now (accepted all certs, including ISRG Root X1 CA):

# dpkg-reconfigure ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Processing triggers for ca-certificates (20211016ubuntu0.16.04.1sav0) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.

Finally:

# wget https://valid-isrgrootx1.letsencrypt.org/
--2023-08-09 18:24:41-- https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org (valid-isrgrootx1.letsencrypt.org)... 52.9.173.94
Connecting to valid-isrgrootx1.lets...

[Mariusz Matuszek (mariuszxc)]

I followed up with a bit of search. I located the ISRG X1 certificate:
ISRG Root X1 CA: https://crt.sh/?id=3958242236

downloaded the 3958242236.crt, then 'cp 3958242236.crt /usr/share/ca-certificates'

# dpkg-reconfigure ca-certificates

(selected the 3958242236.cr for import and imported all)
then

# wget https://valid-isrgrootx1.letsencrypt.org/
--2023-08-09 18:43:38-- https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org (valid-isrgrootx1.letsencrypt.org)... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org (valid-isrgrootx1.letsencrypt.org)|52.9.173.94|:443... connected.
ERROR: cannot verify valid-isrgrootx1.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Unable to locally verify the issuer's authority.

which seemed kind of silly, but I located the chain of certification:
https://crt.sh/?graph=3958242236&opt=nometadata

and followed the import procedure for https://crt.sh/?id=8395

again tested with wget and again test failed.

Now I am stumped.

[Seth Arnold (seth-arnold)]

(I'm not on the livepatch team) I suspect an apt upgrade has gone sideways somewhere along the way and you've got a partially-configured system.

Try:

sudo apt update
sudo apt install -f
sudo dpkg --configure -a
sudo apt upgrade

If I'm right, one of these steps might run a bunch of maintainer scripts. If I'm wrong, probably only the last command would install any updates that aren't yet installed.

Thanks

[Mariusz Matuszek (mariuszxc)]

Hello Seth,

Thanks for the input. Sadly, none of your suggestions worked. See transcript:

# apt update
Get:1 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [97,4 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [99,8 kB]
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [99,8 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [93,7 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [6600 B]
Get:7 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [3328 B]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 DEP-11 Metadata [2464 B]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [130 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [5956 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [326 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [281 kB]
Fetched 1146 kB in 2s (422 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

# apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# dpkg --configure -a

# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
#
# You can verify the status of security fixes using the `pro fix` command.
# E.g., a recent Ruby vulnerability can be checked with: `pro fix USN-6219-1`
# For more detail see: https://ubuntu.com/security/notices/USN-6219-1
#
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[Mariusz Matuszek (mariuszxc)]

Now, this is a long shot, but..

It seems that I have the CA certificate installed, yet wget complains about being unable to validate the web site certificate. The assumption also is, that apt-helper fails for the same reason.

Could it be, that cert verification fails because of some incompatibility between crypto algorithms?
Like, the algorithm needed to verify the certificate being unavailable on my system?

[Mariusz Matuszek (mariuszxc)]

Further test.

I opened https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/ with a browser and displayed page info. In security, Technical Details section I see:

"Connection Encrypted (TLS_AES_256_GCM_SHA384, 256 bit keys, TLS 1.3)"

How can I verify whether this level of encryption is supported with my installed apt-helper and libs?

# ldd /usr/lib/apt/apt-helper
 linux-vdso.so.1 => (0x00007ffd02b92000)
 libapt-pkg.so.5.0 => /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0 (0x00007f962a733000)
 libapt-private.so.0.0 => /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0 (0x00007f962a4d0000)
 libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f962a0ec000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9629ed4000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9629b0a000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9629906000)
 libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f96296eb000)
 libz.so.1 => /usr/local/lib/libz.so.1 (0x00007f96294d0000)
 libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f96292c0000)
 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9629093000)
 liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f9628e70000)
 libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f9628bb3000)
 libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f962ae21000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f96288aa000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f962acb2000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f962868d000)
 libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f962846b000)
 librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f9628263000)
 libgcrypt.so.20 => /usr/local/lib/libgcrypt.so.20 (0x00007f9627f46000)
 libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f9627cd4000)
 libgpg-error.so.0 => /usr/local/lib/libgpg-error.so.0 (0x00007f9627aae000)

apt-cache policy libgcrypt20
libgcrypt20:
  Installed: 1.10.2-2ubuntu1~16.04.sav1
  Candidate: 1.10.2-2ubuntu1~16.04.sav1
  Version table:
 *** 1.10.2-2ubuntu1~16.04.sav1 100
        100 /var/lib/dpkg/status
     1.6.5-2ubuntu0.6 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     1.6.5-2 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

Lastly, i tested with curl:

curl -v https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release -o /tmp/release
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 185.125.190.24:443...
* Connected to esm.ubuntu.com (185.125.190.24) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1...

[Mariusz Matuszek (mariuszxc)]

Following my suspicion about inadequate support for modern cipher suite in apt-helper (or its https method) I started looking for apt backports and located https://launchpad.net/~savoury1/+archive/ubuntu/apt-xenial

After adding this repo and applying updates I tested 'pro enable' again:

# pro enable esm-apps
One moment, checking your subscription first
Updating package lists
Ubuntu Pro: ESM Apps enabled

# pro enable esm-infra
One moment, checking your subscription first
Updating package lists
Ubuntu Pro: ESM Infra enabled

# pro enable livepatch
One moment, checking your subscription first
Stderr: error executing config: livepatchd error: snappy kernel-module-control interface not connected

Stdout:
Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: error executing config: livepatchd error: snappy kernel-module-control interface not connected

ERROR: Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: error executing config: livepatchd error: snappy kernel-module-control interface not connected

So it seems to have solved the problem, except for livepatch.
Will see if if succeeded after kernel upgrade. Currently I have:

# uname -a
Linux xxxx 4.15.0-142-generic #146~16.04.1-Ubuntu SMP Tue Apr 13 09:27:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[Kian Parvin (kian-parvin)]

Hi,

So on the Livepatch issue, the message "snappy kernel-module-control interface not connected" indicates that the Livepatch snap doesn't have the correct permissions from the Snap Daemon. You can view it's slots and plugs (https://snapcraft.io/docs/interface-management) with `snap connections canonical-livepatch`, that output will be helpful for starters.

Could we also get the output from
`snap info canonical-livepatch`
`snap version`

What might also be helpful is to remove and re-install the Livepatch client with `sudo snap remove canonical-livepatch` and `sudo snap install canonical-livepatch`

[Mariusz Matuszek (mariuszxc)]

Hello Kian,

`snap connections canonical-livepatch` reports:
Interface Plug Slot Notes
hardware-observe canonical-livepatch:hardware-observe - -
kernel-module-control canonical-livepatch:kernel-module-control - -
network-bind canonical-livepatch:network-bind - -
network-control canonical-livepatch:network-control - -
network-manager canonical-livepatch:network-manager - -
system-files canonical-livepatch:etc-update-motd-d - -
system-files canonical-livepatch:hostfs-var-local-canonical-livepatch-mode - -
system-observe canonical-livepatch:system-observe - -
system-packages-doc canonical-livepatch:system-packages-doc - -

`snap info canonical-livepatch` shows:
name: canonical-livepatch
summary: Canonical Livepatch Client
publisher: Canonical✓
store-url: https://snapcraft.io/canonical-livepatch
contact: <email address hidden>
license: unset
description: |
  Canonical Livepatch Client
commands:
  - canonical-livepatch
services:
  canonical-livepatch.canonical-livepatchd: simple, enabled, inactive
snap-id: [redacted]
tracking: latest/stable
refresh-date: yesterday at 00:22 CEST
channels:
  latest/stable: 10.6.0 2023-06-22 (235) 10MB -
  latest/candidate: 10.6.0 2023-06-22 (235) 10MB -
  latest/beta: 10.6.0 2023-06-22 (235) 10MB -
  latest/edge: 10.6.0 2023-06-20 (235) 10MB -
  core22/stable: 10.5.7 2023-05-23 (232) 10MB -
  core22/candidate: 10.5.7 2023-05-23 (232) 10MB -
  core22/beta: 10.5.7 2023-05-23 (232) 10MB -
  core22/edge: 10.6.0 2023-06-23 (238) 10MB -
  core20/stable: 10.5.7 2023-05-23 (231) 10MB -
  core20/candidate: 10.5.7 2023-05-23 (231) 10MB -
  core20/beta: 10.5.7 2023-05-23 (231) 10MB -
  core20/edge: 10.6.0 2023-06-20 (237) 10MB -
  core18/stable: 10.5.7 2023-05-23 (230) 10MB -
  core18/candidate: 10.5.7 2023-05-23 (230) 10MB -
  core18/beta: 10.5.7 2023-05-23 (230) 10MB -
  core18/edge: 10.6.0 2023-06-20 (236) 10MB -
  core/stable: 10.5.7 2023-05-23 (229) 10MB -
  core/candidate: 10.5.7 2023-05-23 (229) 10MB -
  core/beta: 10.5.7 2023-05-23 (229) 10MB -
  core/edge: 10.6.0 2023-06-20 (235) 10MB -
installed: 10.6.0 (235) 10MB -

`snap version` output:
snap 2.59.5
snapd 2.59.5
series 16
ubuntu 16.04
kernel 4.15.0-214-generic

Will follow with remove/reinstall.

[Mariusz Matuszek (mariuszxc)]

Following on remove/reinstall:

$ sudo snap remove canonical-livepatch
canonical-livepatch removed

$ sudo snap install canonical-livepatch
canonical-livepatch 10.6.0 from Canonical✓ installed

$ sudo pro enable livepatch
One moment, checking your subscription first
Canonical livepatch enabled.

:-))

Thank you all very much for helpful hints and guidance.

[Grant Orndorff (orndorffgrant)]

I'm glad you got everything working!

Since this turned out to be issues with the environment, I'll mark the bug as Invalid.

Also, not directly related, but since this bug was about enabling ESM, please be aware that packages from https://launchpad.net/~savoury1 are likely to have higher version numbers than those from Ubuntu and therefore will take priority over any ESM updates for those packages. That means that for packages installed from https://launchpad.net/~savoury1 you are relying solely on https://launchpad.net/~savoury1 for any security updates - not Canonical. You are of course free to do this, but this seems like a relevant disclaimer since the bug is about enabling ESM and through your posts it is clear that you have at least ca-certificates, libgcrypt20, and apt from https://launchpad.net/~savoury1

[Mariusz Matuszek (mariuszxc)]

Hello Grant,

I am not going to challenge your decision as you have better feel for the
bug validity rules than me, but I think that
"Since this turned out to be issues with the environment, I'll mark the bug
as Invalid." is not a fair assessment.

If my understanding is correct, the root cause for the problem was
incompatibility between cipher suites offered/required by Ubuntu esm server
and those supported by apt-helper, which ultimately led to rather
misleading error messages. My successful test with accessing the ESM server
with both Firefox and curl proved that my system had the right certificates
and trusted the server certificate *when it could verify it*. Therefore, I
would argue that my _environment_ was correct and the root cause was lack
of support for currently deployed ciphers in the old version of apt-helper.

Please note that apt-helper (and wget, which also failed) are linked
against different crypto libraries than curl and firefox (which both
succeeded).

So I would say this particular interaction was one test case missed by the
QA team.

Anyway, if you can mark the bug as 'wontfix' or such, but make its report
and resulting commentary visible, the whole debugging process could be of
use to a next person with such a problem, googling for answers.

Best Regards
Mariusz

On Thu, Aug 10, 2023 at 8:15 PM Grant Orndorff <email address hidden>
wrote:

> I'm glad you got everything working!
>
> Since this turned out to be issues with the environment, I'll mark the
> bug as Invalid.
>
> Also, not directly related, but since this bug was about enabling ESM,
> please be aware that packages from https://launchpad.net/~savoury1 are
> likely to have higher version numbers than those from Ubuntu and
> therefore will take priority over any ESM updates for those packages.
> That means that for packages installed from
> https://launchpad.net/~savoury1 you are relying solely on
> https://launchpad.net/~savoury1 for any security updates - not
> Canonical. You are of course free to do this, but this seems like a
> relevant disclaimer since the bug is about enabling ESM and through your
> posts it is clear that you have at least ca-certificates, libgcrypt20,
> and apt from https://launchpad.net/~savoury1
>
> ** Changed in: canonical-livepatch-client
> Status: New => Invalid
>
> ** Changed in: ubuntu-advantage-tools (Ubuntu)
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2030866
>
> Title:
> pro attaches but fails to enable services
>
> Status in Canonical Livepatch Client:
> Invalid
> Status in ubuntu-advantage-tools package in Ubuntu:
> Invalid
>
> Bug description:
> # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.7 LTS"
>
>
> # pro attach [REDACTED]
> Enabling default service esm-apps
> Enabling default service esm-infra
> Enabling default service livepatch
> Stderr: error executing config: livepatchd error: snappy
> kernel-module-control interface not connected
>
> Stdout:
> Unable to configure Livepatch: Failed...

[Grant Orndorff (orndorffgrant)]

Hello Mariusz,

I appreciate your concerns and I'm glad you are voicing them here. I also apologize for my terse assessment that is a bit misleading in hindsight. I hope this message helps explain the decision.

First, I assure you that we QA Pro on 16.04/xenial regularly, and it works in a standard xenial setup with all packages updated from Ubuntu. You can test that the particular problem here does not occur in the 16.04 version of apt-helper by testing in a container:

> lxc launch ubuntu:16.04 test-16
Creating test-16
Starting test-16
> lxc exec test-16 -- /usr/lib/apt/apt-helper download-file https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release /tmp/test
Get:1 https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release [6639 B]
Fetched 6639 B in 0s (38.6 kB/s)

or a VM for even higher confidence:

> multipass launch xenial -n test-16-vm
Launched: test-16-vm
> multipass exec test-16-vm -- /usr/lib/apt/apt-helper download-file https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release /tmp
/test
Get:1 https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/Release [6,639 B]
Fetched 6,639 B in 0s (15.6 kB/s)

The root cause of your problem did seem to be related to cipher suite support, but we also determined that your system had some critical packages (including libgcrypt20) installed from https://launchpad.net/~savoury1 (libgcrypt20: Installed: 1.10.2-2ubuntu1~16.04.sav1) which means it is not in a standard setup supported as Ubuntu.

The combination of:
- non-standard crypto packages are installed on the problematic system
- everything works when standard packages are installed on a fresh system
is a signal to me that there is not something within the scope of Ubuntu to fix.
So I would like to keep the state Invalid.

If you can reproduce the issue on a 16.04 system with only standard Ubuntu packages installed, please do re-open the bug.

Installing more packages (apt and dependencies) from https://launchpad.net/~savoury1 seemed to have fixed the problem for you, and that may help others. This bug is Public. Marking the bug Invalid does not prevent others from finding this thread and learning from it.

I hope this satisfies your concerns,
Grant