add-disk action results in failure to import clear_unit_upgrading

[Xav Paice]

Charm cs:ceph-osd-275, xenial.

I tried to add some disks after needing to zap-disk them.

Running the action failed immediately, with the following traceback in the unit log:

2019-02-07 04:54:37 DEBUG juju.worker.uniter agent.go:17 [AGENT-STATUS] executing: running action add-disk
2019-02-07 04:54:37 DEBUG add-disk Traceback (most recent call last):
2019-02-07 04:54:37 DEBUG add-disk File "/var/lib/juju/agents/unit-ceph-osd-18/charm/actions/add-disk", line 27, in <module>
2019-02-07 04:54:37 DEBUG add-disk import ceph_hooks
2019-02-07 04:54:37 DEBUG add-disk File "hooks/ceph_hooks.py", line 96, in <module>
2019-02-07 04:54:37 DEBUG add-disk from charmhelpers.contrib.openstack.utils import (
2019-02-07 04:54:37 DEBUG add-disk ImportError: cannot import name 'clear_unit_upgrading'
2019-02-07 04:54:37 DEBUG juju.worker.uniter.operation executor.go:90 committing operation "run action 3df6a36c-9b63-4ed1-8802-dfd9b619cbf7"
2019-02-07 04:54:37 DEBUG juju.machinelock machinelock.go:180 machine lock released for uniter (run action 3df6a36c-9b63-4ed1-8802-dfd9b619cbf7)

[James Page (james-page)]

I can see from the commit for this charm revision:

$ git status
HEAD detached at e369a09
nothing to commit, working tree clean

$ fgrep -r clear_unit_upgrading *
Binary file hooks/__pycache__/ceph_hooks.cpython-36.pyc matches
hooks/ceph_hooks.py: clear_unit_upgrading,
hooks/ceph_hooks.py: clear_unit_upgrading()
hooks/charmhelpers/contrib/openstack/utils.py:def clear_unit_upgrading():
hooks/charmhelpers/contrib/openstack/utils.py: clear_unit_upgrading()

that the in-tree version of charm-helpers does have the required function.

Is it possible that you have a reactive charm with an older charmhelpers revision that's not using a virtualenv (which is the default in later reactive/layered build versions).

[James Page (james-page)]

(fwiw any charmhelpers version in /usr/local/lib will override the in-charm version of charmhelpers)

[Launchpad Janitor (janitor)]

[Expired for OpenStack ceph-osd charm because there has been no activity for 60 days.]

[Wouter van Bommel (woutervb)]

I bumped into the same problem with release cs:278

There are no charmhelpers in /usr/lib, could not find any trace of it, so that could not be the cause of this problem.

[Alex Kavanagh (ajkavanagh)]

@Wouter, please could you add a few more details?

1. Is this a new install of the charm or an upgrade from a previous version?
2. What other charms are on the same host as the ceph-osd?
3. Can you do a "find . -iname '*.pyc'" in the ceph-osd charm directory on the host with the issue?
4. Can you do "fgrep -r clear_unit_upgrading *" in the ceph-osd charm directory on the host with the issue?

However, the "clear-out stale compiled python files" when into the stable/17.11 charm, and

ceph-osd:275 - UploadTime: "2018-12-05T14:03:20.999Z"
ceph-osd:278 - UploadTime: "2019-03-19T15:43:29.545Z"

which both pre-date that. It is worth checking in /usr/local/lib for any previous versions of charm-helpers.

Thanks.

[Wouter van Bommel (woutervb)]

1. I was with an upgraded charm, not sure how long it's around
2. The following charms are installed:
  * ceilometer-agent
  * filebeat
  * hw-health
  * landscape-client
  * lldpd
  * neutron-openvswitch
  * nrpe-compute
  * ntp
  * nvme-ceph-osd
  * nvme-nova-compute-kvm
  * policy-routing
  * telegraf
3. find . -iname '*.pyc'
./charm/hooks/charmhelpers/contrib/storage/linux/__pycache__/ceph.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/storage/linux/__pycache__/utils.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/storage/linux/__pycache__/loopback.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/storage/linux/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/storage/linux/__pycache__/lvm.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/storage/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/charmsupport/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/charmsupport/__pycache__/nrpe.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/audits/__pycache__/file.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/audits/__pycache__/apt.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/audits/__pycache__/apache.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/audits/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/__pycache__/templating.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/__pycache__/utils.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/__pycache__/harden.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/minimize_access.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/apt.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/profile.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/limits.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/suid_sgid.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/login.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/securetty.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/pam.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/sysctl.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/host/checks/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/ssh/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/ssh/checks/__pycache__/config.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/ssh/checks/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/apache/__pycache__/__init__.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/apache/checks/__pycache__/config.cpython-35.pyc
./charm/hooks/charmhelpers/contrib/hardening/apache/checks/__pycache__/__init__.cpython-35.pyc
....

[Alex Kavanagh (ajkavanagh)]

@Wouter, thanks for the detailed information.

The good news is that there are no stale pyc files anywhere. The bad news is that /usr/local/lib/ charm-helpers package; it's almost certainly the culprit here. How it got there, is open to question. I've looked at all the charms you mentioned and none of them seem to install to that location; but that's the current versions. charmhelpers-0.18.4 is from Jan 19, 2018, so it could be a previous revision of one of the charms that installed it.

If you remove that file (perhaps rename it, in case it breaks an existing charm), does the command now work ... and does it affect any existing charms on the host. Obviously, you may NOT want to do this if it's on a production system, so please don't feel compelled to!

TRIAGE:

For defensive purposes, perhaps, in the charm, we can ensure that /usr/local/lib is demoted in the PATH to ensure that the charm local version is picked up first? This is really about ensuring that the charm doesn't bleed into the system, and vice versa. Obviously, the apt-pkg issue is still there (although we should consider working around that in charm-helpers as well).

[Drew Freiberger (afreiberger)]

I ran into this as well and I found the following interesting:

ubuntu@hostname:/var/lib/juju/agents$ find .|grep wheelhouse|grep help
./unit-os-comp-7/charm/wheelhouse/charmhelpers-0.18.4.tar.gz
./unit-telegraf-58/charm/wheelhouse/charmhelpers-0.19.1.tar.gz
./unit-canonical-livepatch-76/charm/wheelhouse/charmhelpers-0.19.4.tar.gz
./unit-filebeat-56/charm/wheelhouse/charmhelpers-0.19.3.tar.gz

IIUC, the /usr/local/lib/py* bits are from wheelhouses being unpacked, yes?

If so, the issue is that the ubuntu charm on this host (os-comp/7) is holding an incompatible version of charmhelpers.

Also, if this is truly how wheelhouse works, we need to make the wheelhouses unpack relative to the charm agent directory, which may lead us to a juju bug.

[Drew Freiberger (afreiberger)]

workaround suggested above works:

juju run <unit> 'mv /usr/local/lib/python3.5/dist-packages/charmhelpers /usr/local/lib/python3.5/dist-packages/charmhelpers_temp'
juju run-action --wait add-disk .....
juju run <unit> 'mv /usr/local/lib/python3.5/dist-packages/charmhelpers_temp /usr/local/lib/python3.5/dist-packages/charmhelpers'

[Drew Freiberger (afreiberger)]

I didn't find anything related to /usr/local/python3.5/dist-packages in /var/lib/dpkg/info/*.list, so it's not from a package. I think the wheelhouse theory is solid.

[Drew Freiberger (afreiberger)]

I have filed the following against layer-basic:

https://github.com/juju-solutions/layer-basic/issues/141

[Stamatis Katsaounis (skatsaounis)]

Hit that again on stable 20.02 version. Exact same problem, exact same old charm helpers version, exact same workaround.

The only difference is the python version, which is python3.6

[David O Neill (dmzoneill)]

I seem to have run into additional problems

I did the zapping manually

root@ADA345-25-Compute-9:/home/ubuntu# ceph-volume lvm zap /dev/bcache/by-label/bcache-sdb
--> Zapping: /dev/bcache/by-label/bcache-sdb
--> --destroy was not specified, but zapping a whole device will remove the partition table
Running command: dd if=/dev/zero of=/dev/bcache/by-label/bcache-sdb bs=1M count=10
 stderr: 10+0 records in
10+0 records out
 stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120741 s, 868 MB/s
--> Zapping successful for: <Raw Device: /dev/bcache/by-label/bcache-sdb>
root@ADA345-25-Compute-9:/home/ubuntu# ceph-volume lvm zap /dev/bcache/by-label/bcache-sdc
--> Zapping: /dev/bcache/by-label/bcache-sdc
--> --destroy was not specified, but zapping a whole device will remove the partition table
Running command: dd if=/dev/zero of=/dev/bcache/by-label/bcache-sdc bs=1M count=10
 stderr: 10+0 records in
10+0 records out
 stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129391 s, 810 MB/s
--> Zapping successful for: <Raw Device: /dev/bcache/by-label/bcache-sdc>
root@ADA345-25-Compute-9:/home/ubuntu# ceph-volume lvm zap /dev/bcache/by-label/bcache-sdd
--> Zapping: /dev/bcache/by-label/bcache-sdd
--> --destroy was not specified, but zapping a whole device will remove the partition table
Running command: dd if=/dev/zero of=/dev/bcache/by-label/bcache-sdd bs=1M count=10
 stderr: 10+0 records in
10+0 records out
 stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0177458 s, 591 MB/s
--> Zapping successful for: <Raw Device: /dev/bcache/by-label/bcache-sdd>

mv /usr/local/lib/python3.6/dist-packages/charmhelpers /usr/local/lib/python3.6/dist-packages/charmhelpers_temp

juju run-action ceph-osd/30 --wait add-disk osd-devices='/dev/bcache0 /dev/bcache2 /dev/bcache3'

mv /usr/local/lib/python3.6/dist-packages/charmhelpers_temp /usr/local/lib/python3.6/dist-packages/charmhelpers

[James Page (james-page)]

I'm marking this bug as Invalid for ceph-osd - its not installing anything to /usr/local - whatever charm is managing to install to this location needs to be updated to install into a virtualenv or other appropriate non-system path.