rebuild of ceph-fs fails func tests (status set failed)

[Ryan Beisner]

The ceph-fs func tests consistently time out at 5400s (!) in what appears to be an install loop.

I am 3-for-3, hitting the same condition:

ceph-fs/0* maintenance idle 0 172.17.114.10 Installing btrfs-tools,ceph,ceph-mds,gdisk,ntp,python-ceph,python3-pyxattr,xfsprogs

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph-fs/607998/2/6335/index.html

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph-fs/607998/2/6335/test_charm_amulet_smoke_8095/juju-status.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph-fs/607998/2/6335/test_charm_amulet_smoke_8095/amulet-smoke.txt

2018-10-04 19:04:18 INFO juju-log Initializing Apt Layer
2018-10-04 19:04:18 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:04:18 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:04:18 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:08:29 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:08:29 INFO juju-log Initializing Apt Layer
2018-10-04 19:08:29 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:08:30 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:08:30 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:13:20 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:13:20 INFO juju-log Initializing Apt Layer
2018-10-04 19:13:20 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:13:21 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:13:21 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:18:40 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:18:40 INFO juju-log Initializing Apt Layer
2018-10-04 19:18:40 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:18:40 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:18:41 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:23:42 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:23:42 INFO juju-log Initializing Apt Layer
2018-10-04 19:23:42 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:23:42 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:23:42 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:28:47 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:28:47 INFO juju-log Initializing Apt Layer
2018-10-04 19:28:47 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:28:48 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:28:48 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:34:37 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:34:37 INFO juju-log Initializing Apt Layer
2018-10-04 19:34:37 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:34:37 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:34:37 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:40:21 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:40:21 INFO juju-log Initializing Apt Layer
2018-10-04 19:40:21 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:40:21 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:40:22 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:45:49 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:45:49 INFO juju-log Initializing Apt Layer
2018-10-04 19:45:50 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:45:50 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:45:50 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:51:26 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:51:27 INFO juju-log Initializing Apt Layer
2018-10-04 19:51:27 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:51:27 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:51:27 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 19:56:18 INFO juju-log Reactive main running for hook update-status
2018-10-04 19:56:19 INFO juju-log Initializing Apt Layer
2018-10-04 19:56:19 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 19:56:19 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 19:56:19 INFO juju-log status-set failed: active Unit is ready (1 MDS)
2018-10-04 20:00:46 INFO juju-log Reactive main running for hook update-status
2018-10-04 20:00:46 INFO juju-log Initializing Apt Layer
2018-10-04 20:00:46 INFO juju-log Invoking reactive handler: reactive/ceph_fs.py:85:config_changed
2018-10-04 20:00:47 INFO juju-log Invoking reactive handler: reactive/apt.py:49:ensure_package_status
2018-10-04 20:00:47 INFO juju-log status-set failed: active Unit is ready (1 MDS)

[Liam Young (gnuoy)]

I think the charm was hitting https://github.com/stub42/layer-apt/issues/3 which has now been resolved. I have rebuilt the charm and rerun the test and did not hit the issue.