ua attach command fails

[Alex Trepca]

ua attach command fails with:

$ sudo ua attach $TOKEN
Enabling default service esm-infra
Updating package lists
UA Infra: ESM enabled
A reboot is required to complete install.
Enabling default service livepatch
Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/04 02:06:36 error executing config: Livepatchd error: Couldn't setup system logger.

ERROR: Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/04 02:06:36 error executing config: Livepatchd error: Couldn't setup system logger.

Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools
Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

The ubuntu-advantage.log file has this:

2021-05-04 02:06:36,492 - util.py:(414) [DEBUG]: Reading file: /var/lib/cloud/data/result.json
2021-05-04 02:06:36,492 - cli.py:(1035) [ERROR]: Unhandled exception, please file a bug
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 666, in _attach_with_token
    cfg, token, allow_enable=allow_enable
  File "/usr/lib/python3/dist-packages/uaclient/contract.py", line 387, in request_updated_contract
    orig_entitlements, cfg.entitlements, allow_enable
  File "/usr/lib/python3/dist-packages/uaclient/contract.py", line 277, in process_entitlements_delta
    raise exceptions.UserFacingError(status.MESSAGE_UNEXPECTED_ERROR)
uaclient.exceptions.UserFacingError: Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 1004, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 1062, in main
    return args.action(args, cfg)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 184, in new_f
    return f(args, cfg)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 147, in new_f
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 130, in new_f
    retval = f(args, cfg, **kwargs)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 768, in action_attach
    cfg, token=args.token, allow_enable=args.auto_enable
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 677, in _attach_with_token
    cfg.status() # Persist updated status in the event of partial attach
  File "/usr/lib/python3/dist-packages/uaclient/config.py", line 575, in status
    response = self._attached_status()
  File "/usr/lib/python3/dist-packages/uaclient/config.py", line 548, in _attached_status
    self._attached_service_status(ent, inapplicable_resources)
  File "/usr/lib/python3/dist-packages/uaclient/config.py", line 502, in _attached_service_status
    ent_status, details = ent.user_facing_status()
  File "/usr/lib/python3/dist-packages/uaclient/entitlements/base.py", line 469, in user_facing_status
    applicability, details = self.applicability_status()
  File "/usr/lib/python3/dist-packages/uaclient/entitlements/base.py", line 263, in applicability_status
    for error_message, functor, expected_result in self.static_affordances:
  File "/usr/lib/python3/dist-packages/uaclient/entitlements/fips.py", line 267, in static_affordances
    static_affordances = super().static_affordances
  File "/usr/lib/python3/dist-packages/uaclient/entitlements/fips.py", line 113, in static_affordances
    cloud_id = get_cloud_type() or ""
  File "/usr/lib/python3/dist-packages/uaclient/config.py", line 735, in new_f
    return f()
  File "/usr/lib/python3/dist-packages/uaclient/clouds/identity.py", line 68, in get_cloud_type
    return get_cloud_type_from_result_file()
  File "/usr/lib/python3/dist-packages/uaclient/clouds/identity.py", line 56, in get_cloud_type_from_result_file
    dsname = result["v1"]["datasource"].split()[0].lower()
AttributeError: 'NoneType' object has no attribute 'split'

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: ubuntu-advantage-tools 27.0~16.04.1
ProcVersionSignature: Ubuntu 4.4.0-145.171-generic 4.4.176
Uname: Linux 4.4.0-145-generic x86_64
NonfreeKernelModules: falcon_lsm_serviceable falcon_nf_netcontain falcon_kal falcon_lsm_pinned_11611 falcon_lsm_pinned_11312 falcon_lsm_pinned_11110 falcon_lsm_pinned_10807 falcon_lsm_pinned_10703 falcon_lsm_pinned_10404 falcon_lsm_pinned_9918 falcon_lsm_pinned_9808 falcon_lsm_pinned_9804 falcon_lsm_pinned_9705 falcon_lsm_pinned_9607 falcon_lsm_pinned_9510 falcon_lsm_pinned_9403 falcon_lsm_pinned_9402 falcon_lsm_pinned_9205 falcon_lsm_pinned_9101 falcon_lsm_pinned_9003 falcon_lsm_pinned_8706 falcon_lsm_pinned_8705 falcon_lsm_pinned_8703 falcon_lsm_pinned_8701 falcon_lsm_pinned_8406 falcon_lsm_pinned_8404 falcon_lsm_pinned_8402 falcon_lsm_pinned_8202 falcon_lsm_pinned_8101 falcon_lsm_pinned_8002 falcon_lsm_pinned_7901 falcon_lsm_pinned_7803 falcon_lsm_pinned_7603 falcon_lsm_pinned_7602 falcon_lsm_pinned_7503 falcon_lsm_pinned_7403 falcon_lsm_pinned_7304 falcon_lsm_pinned_7303 falcon_lsm_pinned_7203
ApportVersion: 2.20.1-0ubuntu2.30
Architecture: amd64
Date: Tue May 4 02:07:25 2021
Ec2AMI: ami-e1fe2281
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-west-2b
Ec2InstanceType: r3.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubuntu-advantage-tools
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.update-motd.d.88-esm-announce: [deleted]
modified.conffile..etc.update-motd.d.91-contract-ua-esm-status: [deleted]

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Hello Alex, thanks for reporting this bug. If you still have that machine running, can you run:

ua enable livepatch

And verify if the command works. I have one hypothesis here, but if that enable fails as well, then it cannot be what I am thinking.

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Also, can you please provide the livepatch logs by running:

 sudo snap logs canonical-livepatch.canonical-livepatchd

[Casey Marshall (cmars)]

I suspect it might be an issue with snapd not being ready to execute snaps, this might help confirm.

Is this a persistent issue or does it eventually resolve with retries?

Did this happen early in the machine uptime, perhaps in cloud-init?

[Alex Trepca (atrepca0)]

$ sudo ua enable livepatch
One moment, checking your subscription first
Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/04 15:18:29 error executing config: Livepatchd error: Couldn't setup system logger.

ERROR: Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/04 15:18:29 error executing config: Livepatchd error: Couldn't setup system logger.

Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

The error in /var/log/ubuntu-advantage.log is the same.

[Alex Trepca (atrepca0)]

$ sudo snap logs canonical-livepatch.canonical-livepatchd
2021-05-03T23:52:02Z canonical-livepatch.canonical-livepatchd[8596]: 2021/05/03 23:52:02 Couldn't setup system logger.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Main process exited, code=exited, status=1/FAILURE
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Unit entered failed state.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Failed with result 'exit-code'.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Service hold-off time over, scheduling restart.
2021-05-03T23:52:02Z systemd[1]: Stopped Service for snap application canonical-livepatch.canonical-livepatchd.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Start request repeated too quickly.
2021-05-03T23:52:02Z systemd[1]: Failed to start Service for snap application canonical-livepatch.canonical-livepatchd.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Unit entered failed state.
2021-05-03T23:52:02Z systemd[1]: snap.canonical-livepatch.canonical-livepatchd.service: Failed with result 'start-limit-hit'.

[Alex Trepca (atrepca0)]

> Is this a persistent issue or does it eventually resolve with retries?

Persistent issue, doesn't go away.

> Did this happen early in the machine uptime, perhaps in cloud-init?

This is not during the boot process, the instance has been up for a while, I just installed ubuntu-advantage-tools and ran `sudo ua attach $TOKEN` to enable esm_infra.

[Alex Trepca (atrepca0)]

This is happening with more than one instance, it's 10+ 16.04 servers that were all launched around the same time. I'm running the Ansible code again to get an exact count, and look for any similarities.

[Domas Monkus (tasdomas)]

Hello,
we need more information to debug this.

Please include the output from the following commands in your report as plain text.

snap info canonical-livepatch
canonical-livepatch status --verbose
lsb_release -a
uname -a
journalctl -t canonical-livepatch
snap connections
dmesg

Thank you.

Domas Monkus

[Alex Trepca (atrepca0)]

snap info canonical-livepatch:
----------------------------
$ snap info canonical-livepatch
name: canonical-livepatch
summary: Canonical Livepatch Client
publisher: Canonical✓
store-url: https://snapcraft.io/canonical-livepatch
contact: <email address hidden>
license: unset
description: |
  Canonical Livepatch Client
commands:
  - canonical-livepatch
services:
  canonical-livepatch.canonical-livepatchd: simple, enabled, inactive
snap-id: <ID_REMOVED>
tracking: latest/stable
refresh-date: yesterday at 23:51 UTC
channels:
  latest/stable: 9.6.1 2021-04-16 (98) 9MB -
  latest/candidate: 9.6.0 2021-04-12 (97) 9MB -
  latest/beta: 9.6.0 2021-04-12 (97) 9MB -
  latest/edge: 9.6.1 2021-04-14 (98) 9MB -
installed: 9.6.1 (98) 9MB -
----------------------------

canonical-livepatch status --verbose:
----------------------------
$ canonical-livepatch status --verbose
2021/05/04 17:37:37 error executing status: Livepatchd error: Couldn't setup system logger.
----------------------------

lsb_release -a:
----------------------------
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
----------------------------

uname -a:
----------------------------
$ uname -a
Linux <HOSTNAME_REMOVED> 4.4.0-145-generic #171-Ubuntu SMP Tue Mar 26 12:43:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
----------------------------

journalctl -t canonical-livepatch:
----------------------------
$ sudo journalctl -t canonical-livepatch
-- No entries --
----------------------------

snap connections:
----------------------------
$ snap connections
Interface Plug Slot Notes
hardware-observe canonical-livepatch:hardware-observe :hardware-observe -
kernel-module-control canonical-livepatch:kernel-module-control :kernel-module-control -
network-bind canonical-livepatch:network-bind :network-bind -
network-control canonical-livepatch:network-control :network-control -
network-manager canonical-livepatch:network-manager :network-manager -
system-observe canonical-livepatch:system-observe :system-observe -
----------------------------

[Alex Trepca (atrepca0)]

dmesg has a ton of audit messages, no errors of any kind.

[Domas Monkus (tasdomas)]

A couple of follow-up questions:
1. This is running on ec2, correct? Which AMI was used?
2. How was ubuntu-advantage installed?
3. Is there anything else that is non-standard about the system?

[Alex Trepca (atrepca0)]

1. Correct. It's 10 EC2 instances of different types, and using a few different Xenial AMIs:
- ami-e1fe2281 20160921
- ami-a9d276c9 20161020
- ami-7ac6491a 20170307

To make it more confusing, other instances using these same AMIs are fine.

2. It was installed following the steps here: https://discourse.ubuntu.com/t/ubuntu-advantage-client/21788:
- Add ppa:ua-client/stable
- Install ubuntu-advantage-tools package

3. Not sure, what do you mean by non standard? I'd say no but let me know if you're looking for anything specifically and I can check.

[Domas Monkus (tasdomas)]

Thank you for the information. Could you also provide the full ubuntu-advantage.log file?

[Domas Monkus (tasdomas)]

Also, please provide the output of:
```
snap --version
```

And please try the following steps:
1. sudo ua disable livepatch
2. sudo snap remove --purge livepatch
3. sudo ua enable livepatch

[Alex Trepca (atrepca0)]

snap --version:
----------------------------
$ snap --version
snap 2.49.2
snapd 2.49.2
series 16
ubuntu 16.04
kernel 4.4.0-145-generic
----------------------------

Tried the steps here:

$ sudo ua disable livepatch
Livepatch is not currently enabled
See: sudo ua status
Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

$ sudo snap remove --purge livepatch
snap "livepatch" is not installed

$ sudo ua enable livepatch
One moment, checking your subscription first
Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/10 18:04:55 error executing config: Livepatchd error: Couldn't setup system logger.

ERROR: Unable to configure Livepatch: Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/10 18:04:55 error executing config: Livepatchd error: Couldn't setup system logger.

Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

[Alex Trepca (atrepca0)]

Attaching the full ubuntu-advantage.log file too.

[Domas Monkus (tasdomas)]

Hello, Alex,
I don't see the attachment.

Domas

[Alex Trepca (atrepca0)]

Trying again...

[Alex Trepca (atrepca0)]

I can't seem to be able to attach, ok to post the full log here? It's 1342 lines.

[Alex Trepca (atrepca0)]

I uploaded it to this Gist: https://gist.githubusercontent.com/atrepca/dede1e2f4b894a444d6ac0f0abeaf5dd/raw/e5877b08940773e01e5ef93596f31ede49cb69e0/ubuntu-advantage.log

[Richard Harding (rharding)]

It looks like an issue parsing the file: /var/lib/cloud/data/result.json

Are you able to look at share those contents as well? If there's sensitive content I can mark this bug security private to assist.

[Alex Trepca (atrepca0)]

$ cat /var/lib/cloud/data/result.json
{
 "v1": {
  "datasource": null,
  "errors": [
   "[Errno 21] Is a directory: '/var/lib/cloud/instance'",
   "[Errno 21] Is a directory: '/var/lib/cloud/instance'",
   "[Errno 21] Is a directory: '/var/lib/cloud/instance'",
   "[Errno 21] Is a directory: '/var/lib/cloud/instance'"
  ]
 }
}

Can confirm that's a directory:

$ ls -ald /var/lib/cloud/instance
drwxr-xr-x 2 root root 4096 Jun 22 2017 /var/lib/cloud/instance

[Alex Trepca (atrepca0)]

More info, instance is an empty directory, and there's instances too, with the subdirs below:

$ ls -al /var/lib/cloud
total 36
drwxr-xr-x 9 root root 4096 Feb 12 2017 .
drwxr-xr-x 52 root root 4096 May 3 23:46 ..
drwxr-xr-x 2 root root 4096 May 14 2019 data
drwxr-xr-x 2 root root 4096 Oct 14 2016 handlers
drwxr-xr-x 2 root root 4096 Feb 12 2017 instance
drwxr-xr-x 4 root root 4096 Feb 12 2017 instances
drwxr-xr-x 6 root root 4096 Oct 14 2016 scripts
drwxr-xr-x 2 root root 4096 Oct 14 2016 seed
drwxr-xr-x 2 root root 4096 Oct 14 2016 sem

$ ls -al /var/lib/cloud/instance
total 8
drwxr-xr-x 2 root root 4096 Feb 12 2017 .
drwxr-xr-x 9 root root 4096 Feb 12 2017 ..

$ ls -al /var/lib/cloud/instances
total 16
drwxr-xr-x 4 root root 4096 Feb 12 2017 .
drwxr-xr-x 9 root root 4096 Feb 12 2017 ..
drwxr-xr-x 5 root root 4096 Feb 12 2017 i-d04c5fc8
drwxr-xr-x 3 root root 4096 Feb 12 2017 iid-datasource-none

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Looking at the logs, I am seeing two distinct errors. One is livepatch issue that we have:

Failed running command '/snap/bin/canonical-livepatch config remote-server=https://livepatch.canonical.com' [exit(1)]. Message: 2021/05/04 15:18:29 error executing config: Livepatchd error: Couldn't setup system logger.

And the other one is related to cloud-init:
  File "/usr/lib/python3/dist-packages/uaclient/clouds/identity.py", line 56, in get_cloud_type_from_result_file
    dsname = result["v1"]["datasource"].split()[0].lower()
AttributeError: 'NoneType' object has no attribute 'split'

What is interesting in this error is that we are calling the get_cloud_type_from_result_file function. We should only be calling this function if we don't have the cloud-id command in the machine. Since we are running on xenial machines, was not expecing that to happen.

So there is something wrong with cloud-init here. Maybe that is related to the snap issue we are seeing.

Can you run the following cloud-init command and attach the logs here please
cloud-init collect-logs

[Alex Trepca (atrepca0)]

There's no collect-logs for the installed cloud-init version:
----------------------------
$ cloud-init collect-logs
usage: cloud-init [-h] [--version] [--file FILES] [--debug] [--force]
                  {init,modules,query,single,dhclient-hook} ...
cloud-init: error: invalid choice: 'collect-logs' (choose from 'init', 'modules', 'query', 'single', 'dhclient-hook')
----------------------------

$ cloud-init --version
cloud-init 0.7.8

[Alex Trepca (atrepca0)]

These log files exist:

$ ll /var/log/cloud*
-rw-r--r-- 1 syslog adm 379779 May 12 03:01 /var/log/cloud-init.log
-rw-r--r-- 1 root root 65452 May 12 03:01 /var/log/cloud-init-output.log

They're filled with "IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'", I can attach them if needed.

[Richard Harding (rharding)]

$ cloud-init --version
cloud-init 0.7.8

Can you give us more background on that image you're using? Cloud-init went to an annual naming convention in 2017 e.g. 17.1, 17.2 and is up to 21.1 currently. That seems to me like you've got a VERY outdated cloud-init on this system? It should have been upgraded through normal security updates on the host.

I'm going to guess that getting that into a better place will help a lot in this situation. We update cloud-init to help make sure it's working well as the clouds update themselves.

[Alex Trepca (atrepca0)]

I installed cloud-init version 21.1-19-gbad84ad4-0ubuntu1~16.04.2 on all the instances that had the issue, and things are now fine. No idea why they were stuck with the old cloud-init version, I'll look into that.

Thanks for the help,

[Alex Trepca (atrepca0)]

I take that back, on to something else:

$ sudo ua detach
Detach will disable the following services:
    esm-infra
    livepatch
Are you sure? (y/N) y
Updating package lists
This machine is now detached.

$ sudo ua attach <REDACTED>
Enabling default service esm-infra
Updating package lists
UA Infra: ESM enabled
Enabling default service livepatch
Canonical livepatch enabled.
Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools
Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

Excerpt from the ubuntu-advantage.log file:

2021-05-13 14:49:53,778 - util.py:(414) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/status.json
2021-05-13 14:49:53,779 - config.py:(357) [DEBUG]: File does not exist: /var/lib/ubuntu-advantage/status.json
2021-05-13 14:49:53,779 - config.py:(357) [DEBUG]: File does not exist: /var/lib/ubuntu-advantage/status.json
2021-05-13 14:49:53,784 - util.py:(414) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/private/machine-token.json
2021-05-13 14:49:53,784 - util.py:(414) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/private/machine-token.json
2021-05-13 14:49:53,885 - util.py:(572) [DEBUG]: Ran cmd: apt-cache policy, rc: 0 stderr: b''
2021-05-13 14:49:53,885 - util.py:(572) [DEBUG]: Ran cmd: apt-cache policy, rc: 0 stderr: b''
2021-05-13 14:49:53,887 - util.py:(414) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/notices.json
2021-05-13 14:49:53,887 - util.py:(414) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/notices.json
2021-05-13 14:49:53,888 - util.py:(646) [DEBUG]: Writing file: /var/lib/ubuntu-advantage/notices.json
2021-05-13 14:49:53,888 - util.py:(646) [DEBUG]: Writing file: /var/lib/ubuntu-advantage/notices.json
2021-05-13 14:49:53,888 - config.py:(672) [DEBUG]: Using UA client configuration file at /etc/ubuntu-advantage/uaclient.conf
2021-05-13 14:49:53,888 - config.py:(672) [DEBUG]: Using UA client configuration file at /etc/ubuntu-advantage/uaclient.conf
2021-05-13 14:49:53,889 - util.py:(414) [DEBUG]: Reading file: /etc/ubuntu-advantage/uaclient.conf
2021-05-13 14:49:53,889 - util.py:(414) [DEBUG]: Reading file: /etc/ubuntu-advantage/uaclient.conf
2021-05-13 14:49:54,176 - contract.py:(274) [ERROR]: Unexpected error processing contract delta for fips: {'entitlement': {'entitled': True, 'affordances': {'architectures': ['amd64', 'ppc64el', 'ppc64le', 's390x', 'x86_64'], 'series': ['xenial', 'bionic']}, 'type': 'fips', 'obligations': {'enableByDefault': False}, 'directives': {'aptKey': '<REMOVED>', 'additionalPackages': ['ubuntu-fips'], 'suites': ['xenial', 'bionic'], 'aptURL': 'https://esm.ubuntu.com/fips'}}}
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/uaclient/contract.py", line 260, in process_entitlements_delta
    series_overrides=series_overrides,
  File "/usr/lib/python3/dist-packages/uaclient/contract.py", line 330, in process_entitlement_delta
    orig_access, deltas, allow_enable=allow_enable
  File "/usr/lib/python3/dist-packages/uaclient/entitlements/repo.py", line 202, in process_c...

[Alex Trepca (atrepca0)]

Hope the long string after "Ran cmd: /snap/bin/canonical-livepatch enable $LONG_STRING" in the log above is not sensitive, if it is can someone please remove it? Just noticed it, sorry.

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Hi Alex, I think this error is happening because cloud-init has not fully run. I think rebooting the machines would be enough to handle that issue.

But if you don't want to do that, you can restart cloud-init manually by running:

DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
cloud-init clean --logs
cloud-init init --local
cloud-init init > /dev/null
cloud-init modules --mode=config
cloud-init modules --mode=final
cloud-init status --wait --long

This will run all of the cloud-init modules. Let me know if that fix it.

Also, regarding the token, you will need to revoke it unfortunately :(

[Alex Trepca (atrepca0)]

How do I revoke the token? I never typed it in, the livepatch enable was probably done as part of the "sudo ua attach $TOKEN" command, where I used our token. I don't see any livepatch token in our Ubuntu Advantage account, just for UA.

[Alex Trepca (atrepca0)]

Tried the cloud-init steps above on an instance and it worked; also restarting a server fixed it, 'ua attach' reports as attached and 'ua status' shows the enabled services.

I'll reach out to support to revoke the existing UA token.

Thanks for the help, this can be closed.