Charm enters error state after trying to run status --all

[Alexander Balderson]

Today SQA had a handful of deployments which failed with ubuntu-advantage charm going into an error state during config-changed hooks shortly after install.

Most of these deployments failed on AWS, and the error from the call is below:

2024-03-20 19:34:32 ERROR unit.ubuntu-advantage/0.juju-log server.go:325 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 210, in <module>
    main(UbuntuAdvantageCharm)
  File "/var/lib/juju/agents/unit-ubuntu-advantage-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ubuntu-advantage-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ubuntu-advantage-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event) # noqa
  File "/var/lib/juju/agents/unit-ubuntu-advantage-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ubuntu-advantage-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 105, in config_changed
    self._handle_subscription_state()
  File "./src/charm.py", line 158, in _handle_subscription_state
    status = get_status_output()
  File "./src/charm.py", line 55, in get_status_output
    output = subprocess.check_output(["ubuntu-advantage", "status", "--all", "--format", "json"])
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ubuntu-advantage', 'status', '--all', '--format', 'json']' returned non-zero exit status 1.

it doesnt seem that there has been any recent updates to the ubuntu-advantage charm for these tests.

In the following crashdump ubuntu-advantage one was the offending unit, but they all seem to be in error by the time the crashdump is called.

SQA is attaching the ubuntu-advantage key at the time of deployment, and all instances are running Jammy.

an example testrun can be found at:
https://solutions.qa.canonical.com/testruns/e5ba6c60-f45f-4a5e-8120-1be35683168c/
with a crashdump for that deployment at:
https://oil-jenkins.canonical.com/artifacts/e5ba6c60-f45f-4a5e-8120-1be35683168c/generated/generated/kubernetes-aws/juju-crashdump-kubernetes-aws-2024-03-20-21.57.49.tar.gz

[Allan Vidal (alnvdl)]

Hi Alexander,

This charm is mostly a wrapper around the Ubuntu Pro client. The error you got suggests something in the client, so we should be looking at /var/log/ubuntu-advantage.log

Looking at the crashdump you sent, this file shows the following error for multiple machines:

["2024-03-20T21:57:23.912", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /var/lib/ubuntu-advantage/lock", {}]
["2024-03-20T21:57:23.912", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call last):\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n return func(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 257, in new_f\n return f(args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 202, in new_f\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 186, in new_f\n retval = f(*args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1305, in action_attach\n actions.attach_with_token(cfg, token=token, allow_enable=allow_enable)\n File \"/usr/lib/python3/dist-packages/uaclient/actions.py\", line 65, in attach_with_token\n new_machine_token = contract_client.add_contract_machine(\n File \"/usr/lib/python3/dist-packages/uaclient/util.py\", line 92, in decorator\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 96, in add_contract_machine\n response = self.request_url(\n File \"/usr/lib/python3/dist-packages/uaclient/http/serviceclient.py\", line 68, in request_url\n return http.readurl(\n File \"/usr/lib/python3/dist-packages/uaclient/http/__init__.py\", line 364, in readurl\n resp = _readurl_urllib(req, timeout=timeout)\n File \"/usr/lib/python3/dist-packages/uaclient/http/__init__.py\", line 167, in _readurl_urllib\n resp = request.urlopen(req, timeout=timeout)\n File \"/usr/lib/python3.10/urllib/request.py\", line 216, in urlopen\n return opener.open(url, data, timeout)\n File \"/usr/lib/python3.10/urllib/request.py\", line 519, in open\n response = self._open(req, data)\n File \"/usr/lib/python3.10/urllib/request.py\", line 536, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n File \"/usr/lib/python3.10/urllib/request.py\", line 496, in _call_chain\n result = func(*args)\n File \"/usr/lib/python3.10/urllib/request.py\", line 1391, in https_open\n return self.do_open(http.client.HTTPSConnection, req,\n File \"/usr/lib/python3.10/urllib/request.py\", line 1352, in do_open\n r = h.getresponse()\n File \"/usr/lib/python3.10/http/client.py\", line 1375, in getresponse\n response.begin()\n File \"/usr/lib/python3.10/http/client.py\", line 318, in begin\n version, status, reason = self._read_status()\n File \"/usr/lib/python3.10/http/client.py\", line 287, i...