ceph-radosgw/0 charm fails install step with "No module named apt" error

[Ed Stewart]

Since earlier today, our CI/CD has started failing deploying the ceph-radosgw/0 juju charm using the following bundle snippet (latest charm from store - same error with radosgw-262 too). We get an install error.

  ceph-radosgw:
    annotations:
      gui-x: '1000'
      gui-y: '250'
    charm: cs:ceph-radosgw-263
    num_units: 1
    options:
      source: cloud:bionic-rocky
    to:
    - lxd:0

Error from juju logs:

2019-02-08 17:34:42 DEBUG install dpkg-query: package 'python-apt' is not installed and no information is available
2019-02-08 17:34:42 DEBUG install Use dpkg --info (= dpkg-deb --info) to examine archive files,
2019-02-08 17:34:42 DEBUG install and dpkg --contents (= dpkg-deb --contents) to list their contents.
2019-02-08 17:34:42 DEBUG install Reading package lists...
2019-02-08 17:34:42 DEBUG install Building dependency tree...
2019-02-08 17:34:42 DEBUG install Reading state information...
2019-02-08 17:34:42 DEBUG install The following package was automatically installed and is no longer required:
2019-02-08 17:34:42 DEBUG install libfreetype6
2019-02-08 17:34:42 DEBUG install Use 'apt autoremove' to remove it.
2019-02-08 17:34:42 DEBUG install Suggested packages:
2019-02-08 17:34:42 DEBUG install python-apt-dbg python-apt-doc
2019-02-08 17:34:42 DEBUG install The following NEW packages will be installed:
2019-02-08 17:34:42 DEBUG install python-apt
2019-02-08 17:34:42 DEBUG install 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
2019-02-08 17:34:42 DEBUG install Need to get 149 kB of archives.
2019-02-08 17:34:42 DEBUG install After this operation, 682 kB of additional disk space will be used.
2019-02-08 17:34:42 DEBUG install Err:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-apt amd64 1.6.3
2019-02-08 17:34:42 DEBUG install 404 Not Found [IP: 91.189.88.161 80]
2019-02-08 17:34:42 DEBUG install E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python-apt/python-apt_1.6.3_amd64.deb 404 Not Found [IP: 91.189.88.161 80]
2019-02-08 17:34:42 DEBUG install E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2019-02-08 17:34:42 DEBUG install dpkg-query: package 'python-pip' is not installed and no information is available
2019-02-08 17:34:42 DEBUG install Use dpkg --info (= dpkg-deb --info) to examine archive files,
2019-02-08 17:34:42 DEBUG install and dpkg --contents (= dpkg-deb --contents) to list their contents.
2019-02-08 17:34:42 DEBUG install Reading package lists...
2019-02-08 17:34:42 DEBUG install Building dependency tree...
2019-02-08 17:34:42 DEBUG install Reading state information...
2019-02-08 17:34:42 DEBUG install E: Unable to locate package python-pip
2019-02-08 17:34:43 DEBUG install Traceback (most recent call last):
2019-02-08 17:34:43 DEBUG install File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/install.real", line 89, in <module>
2019-02-08 17:34:43 DEBUG install from charmhelpers.contrib.hardening.harden import harden
2019-02-08 17:34:43 DEBUG install File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 25, in <module>
2019-02-08 17:34:43 DEBUG install from charmhelpers.contrib.hardening.host.checks import run_os_checks
2019-02-08 17:34:43 DEBUG install File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/host/checks/__init__.py", line 19, in <module>
2019-02-08 17:34:43 DEBUG install from charmhelpers.contrib.hardening.host.checks import (
2019-02-08 17:34:43 DEBUG install File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/host/checks/apt.py", line 16, in <module>
2019-02-08 17:34:43 DEBUG install from charmhelpers.contrib.hardening.audits.apt import (
2019-02-08 17:34:43 DEBUG install File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/audits/apt.py", line 16, in <module>
2019-02-08 17:34:43 DEBUG install from apt import apt_pkg
2019-02-08 17:34:43 DEBUG install ImportError: No module named apt
2019-02-08 17:34:43 ERROR juju.worker.uniter.operation runhook.go:114 hook "install" failed: exit status 1

Model Controller Cloud/Region Version SLA Timestamp
cicd-continuous google-controller google/us-east1 2.4.4 unsupported 18:47:22Z
App Version Status Scale Charm Store Rev OS Notes
central-monitor active 1 nagios jujucharms 28 ubuntu
ceph-mon 13.2.4+dfsg1 active 3 ceph-mon jujucharms 31 ubuntu
ceph-osd 13.2.4+dfsg1 active 3 ceph-osd jujucharms 273 ubuntu
ceph-radosgw error 1 ceph-radosgw jujucharms 262 ubuntu
cinder 13.0.2 active 1 cinder jujucharms 276 ubuntu
cinder-ceph 13.0.2 active 1 cinder-ceph jujucharms 238 ubuntu
glance 17.0.0 active 1 glance jujucharms 271 ubuntu
keystone 14.0.1 active 1 keystone jujucharms 288 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 272 ubuntu
neutron-api 13.0.2 active 1 neutron-api jujucharms 266 ubuntu
neutron-gateway 13.0.2 active 1 neutron-gateway jujucharms 256 ubuntu
neutron-openvswitch 13.0.2 active 1 neutron-openvswitch jujucharms 255 ubuntu
nova-cloud-controller 18.0.3 active 1 nova-cloud-controller jujucharms 316 ubuntu
nova-compute 18.0.3 active 1 nova-compute jujucharms 419 ubuntu
ntp 3.2 active 2 ntp jujucharms 31 ubuntu
openstack-dashboard 14.0.1 active 1 openstack-dashboard local 32 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 82 ubuntu
Unit Workload Agent Machine Public address Ports Message
central-monitor/0* active idle 0/lxd/0 252.1.120.137 80/tcp ready
ceph-mon/0 active idle 0/lxd/1 252.1.116.91 Unit is ready and clustered
ceph-mon/1 active idle 0/lxd/2 252.1.127.220 Unit is ready and clustered
ceph-mon/2* active idle 0/lxd/3 252.1.115.192 Unit is ready and clustered
ceph-osd/0 active idle 0/lxd/4 252.1.127.172 Unit is ready (3 OSD)
ceph-osd/1* active idle 0/lxd/5 252.1.112.238 Unit is ready (3 OSD)
ceph-osd/2 active idle 0/lxd/6 252.1.125.208 Unit is ready (3 OSD)
ceph-radosgw/0* error idle 0/lxd/7 252.1.113.230 hook failed: "install"
cinder/0* active idle 0/lxd/8 252.1.121.167 8776/tcp Unit is ready
  cinder-ceph/0* active idle 252.1.121.167 Unit is ready
glance/0* active idle 0/lxd/9 252.1.126.205 9292/tcp Unit is ready
keystone/0* active idle 0/lxd/10 252.1.123.90 5000/tcp Unit is ready

This is deploying into an lxd container.

Manually trying to download http://archive.ubuntu.com/ubuntu/pool/main/p/python-apt/python-apt_1.6.3_amd64.deb reproduces the same issue.

The package on the archive site seems to have been renamed "python-apt_1.6.3ubuntu1_amd64.deb".

[Ryan Beisner (1chb1n)]

I'm working on reproducing this, I was able to reproduce a similar condition in a charm-upgrade:

2019-02-08 20:39:12 DEBUG upgrade-charm Unpacking python-markupsafe (1.0-1build1) ...
2019-02-08 20:39:12 DEBUG upgrade-charm Selecting previously unselected package python-jinja2.
2019-02-08 20:39:12 DEBUG upgrade-charm Preparing to unpack .../python-jinja2_2.10-1_all.deb ...
2019-02-08 20:39:12 DEBUG upgrade-charm Unpacking python-jinja2 (2.10-1) ...
2019-02-08 20:39:12 DEBUG upgrade-charm Setting up python-markupsafe (1.0-1build1) ...
2019-02-08 20:39:12 DEBUG upgrade-charm Setting up python-jinja2 (2.10-1) ...
2019-02-08 20:39:14 DEBUG upgrade-charm Traceback (most recent call last):
2019-02-08 20:39:14 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/upgrade-charm", line 90, in <module>
2019-02-08 20:39:14 DEBUG upgrade-charm from charmhelpers.contrib.hardening.harden import harden
2019-02-08 20:39:14 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 25, in <module>
2019-02-08 20:39:14 DEBUG upgrade-charm from charmhelpers.contrib.hardening.host.checks import run_os_checks
2019-02-08 20:39:14 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/host/checks/__init__.py", line 19, in <module>
2019-02-08 20:39:14 DEBUG upgrade-charm from charmhelpers.contrib.hardening.host.checks import (
2019-02-08 20:39:14 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/host/checks/apt.py", line 16, in <module>
2019-02-08 20:39:14 DEBUG upgrade-charm from charmhelpers.contrib.hardening.audits.apt import (
2019-02-08 20:39:14 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/hardening/audits/apt.py", line 16, in <module>
2019-02-08 20:39:14 DEBUG upgrade-charm from apt import apt_pkg
2019-02-08 20:39:14 DEBUG upgrade-charm ImportError: No module named apt
2019-02-08 20:39:14 ERROR juju.worker.uniter.operation runhook.go:132 hook "upgrade-charm" failed: exit status 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/635933

[Ryan Beisner (1chb1n)]

Actually, I've discovered that the test host that I was using had a host network/connectivity issue. On a clean run in our lab, I'm not able to reproduce the issue.

If there is a cache/proxy/mirror mechanism in place on your network, please ensure it is up-to-date and healthy.

If the issue persists, please provide the following additional detail:

Juju version
Host version details
Image source details

Related to, but not directly causing this bug, I've raised a gerrit review to improve visibility and earlier and more clear failure mode when the early apt operations fail (for any reason).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/635940

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Beisner (<email address hidden>) on branch: master
Review: https://review.openstack.org/635933

[Andrey Grebennikov (agrebennikov)]

I'm performing clean install right now and it is 100% reproducible.
This is definitely a package name issue:

2019-02-09 23:04:48 DEBUG install Err:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-apt amd64 1.6.3
2019-02-09 23:04:48 DEBUG install 404 Not Found [IP: 91.189.88.152 80]
2019-02-09 23:04:48 DEBUG install E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python-apt/python-apt_1.6.3_amd64.deb 404 Not Found [IP: 91.189.88.152 80]
2019-02-09 23:04:48 DEBUG install E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2019-02-09 23:04:48 DEBUG install dpkg-query: package 'python-pip' is not installed and no information is available
2019-02-09 23:04:48 DEBUG install Use dpkg --info (= dpkg-deb --info) to examine archive files,
2019-02-09 23:04:48 DEBUG install and dpkg --contents (= dpkg-deb --contents) to list their contents.
2019-02-09 23:04:48 DEBUG install Reading package lists...
2019-02-09 23:04:48 DEBUG install Building dependency tree...
2019-02-09 23:04:48 DEBUG install Reading state information...
2019-02-09 23:04:48 DEBUG install E: Unable to locate package python-pip

[Andrey Grebennikov (agrebennikov)]

Hm... this seems to be somehow related to a wrong execution of apt-get update seems so... When I manually ran apt-get update on the unit - I was able to apt-get install python-apt with no problem.

[Andrey Grebennikov (agrebennikov)]

Confirming - new unit was able to deploy with no issues.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635940
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=c4e5565799fa761494cce96f75e5baca10da290f
Submitter: Zuul
Branch: master

commit c4e5565799fa761494cce96f75e5baca10da290f
Author: Ryan Beisner <email address hidden>
Date: Fri Feb 8 15:10:21 2019 -0600

    Update pre-install hooks to fail on error

    The pre-install operations may fail, yet that failure is not
    elevated to the user. This masks the failure and makes early
    package install issues difficult to troubleshoot.

    If the basic pre-install script fails, the charm should not
    proceed to later hooks as the requirements may not be met.

    Hashbangs for bash should specify -e (errexit) on all of the
    pre-install bash scripts.

    Change-Id: Ie7b99dfa4cbe00a03acf1b2cc2eeecc7f84fbe17
    Closes-bug: #1815243
    Partial-bug: #1815231

[Ed Stewart (emcs2)]

Confirming, this issue went away by Friday evening. Thanks for taking a look. We didn't make any changes our side, and we aren't caching anything ourside (not even ubuntu images).

The file it complainedd about http://archive.ubuntu.com/ubuntu/pool/main/p/python-apt/python-apt_1.6.3_amd64.deb still doesn't exist, so perhaps something in the ubuntu base build needed to be updated in parallel with the rename of the python-apt deb file and we caught it whilst it was out of sync. I'm purely guessing.

[Dmitrii Shcherbakov (dmitriis)]

This seems very similar to https://bugs.launchpad.net/juju/bug/1807615 (see also https://github.com/juju/juju/pull/9751)

Especially if there is a proxy in the environment.