Can't enable or disable livepatch.

[Robin Huber]

I cannot use livepatch and I don't know why. I tried to completely purge the snap but it always results in the cryptic error

connection to the daemon failed: Get "http://127.0.0.1/enable": dial unix /var/snap/canonical-livepatch/278/livepatchd.sock: connect: no such file or directory

Content of lsb_release -rb:

Description: Ubuntu 22.04.4 LTS
Release: 22.04

Canonical-livepatch is a snap, and as such I cannot provide apt-cache policy

Output of uname -a:

Linux loyaltymc 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: ubuntu-advantage-tools 31.2.3~22.04
ProcVersionSignature: Ubuntu 6.5.0-28.29~22.04.1-generic 6.5.13
Uname: Linux 6.5.0-28-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Sun Apr 21 15:26:59 2024
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubuntu-advantage-tools
UpgradeStatus: No upgrade log present (probably fresh install)
apparmor_logs.txt:

cloud-id.txt-error: Invalid command specified 'cloud-id'.
livepatch-status.txt-error: Failed running command '/snap/bin/canonical-livepatch status' [exit(1)]. Message: connection to the daemon failed: Get "http://127.0.0.1/status?format=yaml&verbosity=2": dial unix /var/snap/canonical-livepatch/278/livepatchd.sock: connect: no such file or directory
pro-journal.txt: -- No entries --
uaclient.conf:
 contract_url: https://contracts.canonical.com
 log_level: debug

[Grant Orndorff (orndorffgrant)]

Hello theroblp!

Thank you for the bug report! I'm assigning this bug to the livepatch snap as this seems to be a bug in the snap package.

It looks like even `snap install canonical-livepatch` is failing.

I'll leave it as "incomplete" for the ubuntu-pro-client (ubuntu-advantage-tools) for now in case it turns out to be a bug there.

[Kian Parvin (kian-parvin)]

Hi Robin,

Could you please uninstall the snap with `sudo snap remove canonical-livepatch` then re-install it with `sudo snap install canonical-livepatch` and then provide us with the output from the following commands to investigate further.

- snap info canonical-livepatch
- canonical-livepatch status --verbose
- journalctl -t canonical-livepatch.canonical-livepatchd
- snap connections canonical-livepatch
- ls -la /var/snap/canonical-livepatch/current

The Livepatch snap is made up of a daemon and a client, the daemon is always running in the background and when you run a command like `canonical-livepatch status` the client sends a request over a unix socket located at /var/snap/canonical-livepatch/current/livepatchd.sock or /var/snap/canonical-livepatch/current/livepatchd-priv.sock to the daemon. On your system it looks like the socket is never created.

[Robin Huber (theroblp)]

Hello Kian. Apologies for the long delay, I have temporarily forgotten about this issue ticket. I will provide what you asked for below. Thank you again for your help.

//////////////////////////////
snap info canonical-livepatch
//////////////////////////////

name: canonical-livepatch
summary: Canonical Livepatch Client
publisher: Canonical✓
store-url: https://snapcraft.io/canonical-livepatch
contact: <email address hidden>
license: unset
description: |
  Canonical Livepatch patches high and critical linux kernel vulnerabilities removing the
  immediate need to reboot to upgrade the kernel, instead allowing the downtime to be
  scheduled. It is a part of the Ubuntu Pro offering.

  The Canonical Livepatch Client is an application that runs on your machine and
  periodically checks for patches.

  See our docs at https://ubuntu.com/security/livepatch/docs/livepatch
commands:
  - canonical-livepatch
services:
  canonical-livepatch.canonical-livepatchd: simple, enabled, inactive
snap-id: b96UJ4vttpNhpbaCWctVzfduQcPwQ5wn
tracking: latest/stable
refresh-date: today at 13:28 CEST
channels:
  latest/stable: 10.8.2 2024-04-17 (278) 10MB -
  latest/candidate: 10.8.2 2024-04-17 (278) 10MB -
  latest/beta: 10.8.2 2024-04-17 (278) 10MB -
  latest/edge: 10.8.3 2024-06-04 (282) 10MB -
  core22/stable: 10.8.1 2024-02-06 (264) 10MB -
  core22/candidate: 10.8.1 2024-02-06 (264) 10MB -
  core22/beta: 10.8.1 2024-02-06 (264) 10MB -
  core22/edge: 10.8.1 2024-02-06 (264) 10MB -
  core20/stable: 10.7.0 2023-09-28 (247) 10MB -
  core20/candidate: 10.7.0 2023-09-28 (247) 10MB -
  core20/beta: 10.7.0 2023-09-28 (247) 10MB -
  core20/edge: 10.8.1 2024-02-06 (269) 10MB -
  core18/stable: 10.6.0 2023-09-15 (236) 10MB -
  core18/candidate: 10.6.0 2023-09-15 (236) 10MB -
  core18/beta: 10.6.0 2023-09-15 (236) 10MB -
  core18/edge: 10.8.1 2024-02-06 (268) 10MB -
  core/stable: 10.7.0 2023-09-28 (246) 10MB -
  core/candidate: 10.7.0 2023-09-28 (246) 10MB -
  core/beta: 10.7.0 2023-09-28 (246) 10MB -
  core/edge: 10.8.1 2024-02-06 (267) 10MB -
  bare/stable: –
  bare/candidate: –
  bare/beta: –
  bare/edge: 10.8.0-beta1 2023-11-22 (254) 11MB -
installed: 10.8.2 (278) 10MB -

////////////////////////////////////
canonical-livepatch status --verbose
////////////////////////////////////

connection to the daemon failed: Get "http://127.0.0.1/status?format=json&verbosity=3": dial unix /var/snap/canonical-livepatch/278/livepatchd.sock: connect: no such file or directory

//////////////////////////////////////////////////////
journalctl -t canonical-livepatch.canonical-livepatchd
//////////////////////////////////////////////////////

No journal files were found.
-- No entries --

////////////////////////////////////
snap connections canonical-livepatch
////////////////////////////////////

Interface Plug Slot Notes
hardware-observe ...

[Kian Parvin (kian-parvin)]

Thanks for the info, it seems like the Livepatch Snap didn't install properly because the file at /var/snap/canonical-livepatch/278/livepatchd.sock is missing. I'd recommend trying to remove and reinstall Livepatch, try the following commands.

- sudo snap remove canonical-livepatch
- sudo snap install canonical-livepatch
- pro enable livepatch

Note sure if that last command needs sudo.
Let us know if that fixes the issue otherwise we'll need to dig deeper.

[Robin Huber (theroblp)]

Hello there, and thank you for your reply.

I have already tried the above commands multiple times, even with the --purge flag to wipe clean the configs. I also tried to regenerate my machine-id and then re-install livepatch which didn't work either. I run all commands with sudo (Technically speaking, as I log in with root using "sudo su -l root" for making system changes)

[Kian Parvin (kian-parvin)]

Gotcha,

So let's try a few things and if it's not coming right we may need to rope in some help from Snap developers.

The livepatch daemon, when it starts, creates 2 unix sockets in `/var/snap/canonical-livepatch/current/`. When the daemon is killed/stopped/fails to start, these files should no longer exist. So let's try a few things,

Run the following to watch the logs,
- sudo snap logs canonical-livepatch -f

Now run the following in a separate terminal

- ls -la /var/snap/canonical-livepatch/current/
- sudo snap start canonical-livepatch
- ls -la /var/snap/canonical-livepatch/current/

Do the logs say anything useful?
You can also try the above while watching the logs from `journalctl -f`

[Robin Huber (theroblp)]

No, unfortunately the logs say nothing at all. And there is nothing inside /var/snap/canonical-livepatch/current. Also, I noticed something not sure if it is important, but when I try to autocomplete "sudo snap logs canonical-livepatch" it refers to the name "canonical-livepatch.canonical-livepatchd" instead.

[Kian Parvin (kian-parvin)]

Hmm this is very strange. One last thing worth trying is executing the daemon manually.

First, can you run the following commands and provide the output:

- ps aux | grep livepatch
- sudo snap stop canonical-livepatch
- ps aux | grep livepatch

Then try run the daemon manually

- sudo SNAP_COMMON=/var/snap/canonical-livepatch/common SNAP_DATA=/var/snap/canonical-livepatch/278 /snap/canonical-livepatch/current/canonical-livepatchd

And then in a separate terminal,

- canonical-livepatch status

Let's see if that reveals anything.

[Robin Huber (theroblp)]

Here you go:

///////////////////////
ps aux | grep livepatch
///////////////////////

root 749614 0.0 0.0 17736 6720 ? S 12:37 0:00 journalctl -o json --no-pager -n 10 -f --namespace=* -u snap.canonical-livepatch.canonical-livepatchd.service
root 755884 0.0 0.0 17736 6720 ? S 12:39 0:00 journalctl -o json --no-pager -n 10 -f --namespace=* -u snap.canonical-livepatch.canonical-livepatchd.service
root 756858 0.0 0.0 17736 6720 ? S 12:41 0:00 journalctl -o json --no-pager -n 10 -f --namespace=* -u snap.canonical-livepatch.canonical-livepatchd.service
root 841197 0.0 0.0 6620 2496 pts/19 S+ 15:45 0:00 grep --color=auto livepatch

/////////////////////////////////////////////////
ps aux | grep livepatch (After stopping the snap)
/////////////////////////////////////////////////

Same result as above

Using the snap_common command it states only "Livepatch client disabled." canonical-livepatch status states the same error as before. Could it be that there are two seperate versions of livepatch, maybe and could they interfer with each other? Because in snap I can start/stop a snap called "canonical-livepatch" as well as one called "canonical-livepatch.canonical-livepatchd". Or is that just an alias?

[Kian Parvin (kian-parvin)]

Okay this was very helpful!

The message "Livepatch client disabled." is only printed when Livepatch is set to be disabled. There are 3 ways to disable/modify the "mode" that Livepatch starts to diagnose issues and prevent avoid faulty patches. It looks like your mode is set to completely disable starting the Livepatch daemon.

- Kernel cmdline: Run `cat /proc/cmdline` and look for a key=value pair where the key is "canonical_livepatch_mode"
- Check for a mode file with `cat /var/local/canonical_livepatch_mode` and see if there is anything inside the file.
- Check for a mode file with `cat /var/lib/snapd/hostfs/var/local/canonical_livepatch_mode"` and see if there is anything inside the file.

And for reference, the "canonical-livepatch.canonical-livepatchd" that comes up when using the snap commands is the name of the service inside the snap. Try `snap info canonical-livepatch` and you can see all the services inside the snap (only 1) and you can try the same with other snaps. Services of a snap can be started/stopped, etc. Running `snap start/stop snap.service` will only start/stop the specified service while `snap start/stop snap` will start/stop all services for that snap. More details are here https://snapcraft.io/docs/service-management

[Robin Huber (theroblp)]

//////////////////
cat /proc/cmdline
/////////////////

BOOT_IMAGE=/vmlinuz-6.5.0-28-generic root=UUID=[redacted] ro consoleblank=0 systemd.show_status=true

///////////////////////////////////////
cat /var/local/canonical_livepatch_mode
//////////////////////////////////////

stop

////////////////////////////////////////////////////////////
cat /var/lib/snapd/hostfs/var/local/canonical_livepatch_mode
///////////////////////////////////////////////////////////

cat: /var/lib/snapd/hostfs/var/local/canonical_livepatch_mode: No such file or directory

Also, thanks dor the service reference, that clears some confusion about it.

[Kian Parvin (kian-parvin)]

Nice, that "stop" inside /var/local/canonical_livepatch_mode should be the source of the issue. You can simply delete that file and run `sudo snap restart canonical-livepatch` and things should be well again.

[Robin Huber (theroblp)]

YES!!! It works again.

Thank you so much for all the help, Mr. Parvin. I would have never figured it out by myself. I have no idea how this could have happened. I never touched this file, and I don't remember ever uninstalling or disabling Livepatch in an unusual way before. Weird tough, that snap's --purge flag didn't get rid of this file. Funny to think that 4 little words in a file can cause such a cryptic error. Anyways, I am so glad it works again. Again, thank you very much. I still have a lot to learn in Ubuntu, and I am glad you where there to help me out.

With best Regards
Robin H.