mon-relation-changed IndexError: list index out of range

[Alexandros Soumplis]

Fresh install of a ceph node with some disks reused from a previous ceph installation. Cleared lvs infomration on disks and the zap and add. In ceph everything is ok, but juju complains with:

2019-03-10 22:33:49 DEBUG mon-relation-changed Traceback (most recent call last):
2019-03-10 22:33:49 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-pilot-ceph-osd-20/charm/hooks/mon-relation-changed", line 746, in <module>
2019-03-10 22:33:49 DEBUG mon-relation-changed hooks.execute(sys.argv)
2019-03-10 22:33:49 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-pilot-ceph-osd-20/charm/hooks/charmhelpers/core/hookenv.py", line 909, in execute
2019-03-10 22:33:49 DEBUG mon-relation-changed self._hooks[hook_name]()
2019-03-10 22:33:49 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-pilot-ceph-osd-20/charm/hooks/mon-relation-changed", line 589, in mon_relation
2019-03-10 22:33:49 DEBUG mon-relation-changed prepare_disks_and_activate()
2019-03-10 22:33:49 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-pilot-ceph-osd-20/charm/hooks/mon-relation-changed", line 474, in prepare_disks_and_activate
2019-03-10 22:33:49 DEBUG mon-relation-changed devices = [dev for dev in devices
2019-03-10 22:33:49 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-pilot-ceph-osd-20/charm/hooks/mon-relation-changed", line 475, in <listcomp>
2019-03-10 22:33:49 DEBUG mon-relation-changed if not ceph.is_active_bluestore_device(dev)]
2019-03-10 22:33:49 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1682, in is_active_bluestore_device
2019-03-10 22:33:49 DEBUG mon-relation-changed lv_name = lvm.list_logical_volumes('vg_name={}'.format(vg_name))[0]
2019-03-10 22:33:49 DEBUG mon-relation-changed IndexError: list index out of range
2019-03-10 22:33:49 ERROR juju.worker.uniter.operation runhook.go:132 hook "mon-relation-changed" failed: exit status 1

A little dig further reveals that the function list_logical_volumes runs the command
lvs --options lv_name --noheadings

which returns:

  osd-block-3217d6f3-3754-40fc-8415-dcb5765a20d7
  osd-block-3d1431be-d1ac-4a4c-9a88-27d0651b081c
  osd-block-41d0a39f-53bf-47b2-8be2-da712000c962
  osd-block-42b82149-061b-469c-8f2e-b349ba4d57c7
  osd-block-68146527-cf17-402b-89cb-6730d03abb18
  osd-block-90352921-7392-4495-92ef-5766731b3989
  osd-block-969ad624-1882-4d17-8fe7-3b6aededcebc
  osd-block-a120e31a-4c61-42ed-b902-060338745605
  osd-block-a305970e-8655-4458-b793-ec053fdb2a94
  osd-block-b40e0ce5-2fb9-403f-af75-3618e44870a4
  osd-block-e2828d3e-0f2d-4839-a353-6fcc5d3e7700
  osd-wal-3217d6f3-3754-40fc-8415-dcb5765a20d7
  osd-wal-3d1431be-d1ac-4a4c-9a88-27d0651b081c
  osd-wal-41d0a39f-53bf-47b2-8be2-da712000c962
  osd-wal-42b82149-061b-469c-8f2e-b349ba4d57c7
  osd-wal-68146527-cf17-402b-89cb-6730d03abb18
  osd-wal-90352921-7392-4495-92ef-5766731b3989
  osd-wal-969ad624-1882-4d17-8fe7-3b6aededcebc
  osd-wal-a120e31a-4c61-42ed-b902-060338745605
  osd-wal-a305970e-8655-4458-b793-ec053fdb2a94
  osd-wal-b40e0ce5-2fb9-403f-af75-3618e44870a4
  osd-wal-e2828d3e-0f2d-4839-a353-6fcc5d3e7700
  lvroot

As a temporary workaround I edited hooks/charmhelpers/contrib/storage/linux/lvm.py
124c124
< lvs = []
---
> lvs = ['']

This way the failed hook continued and everything seems to be working as expected

[David Ames (thedac)]

TRIAGE:

In is_active_bluestore_device call to check the return value of list_logical_volumes before attempting to get the zeroth value from the list.

diff --git a/lib/ceph/utils.py b/lib/ceph/utils.py
index 35f351f..220039b 100644
--- a/lib/ceph/utils.py
+++ b/lib/ceph/utils.py
@@ -1692,7 +1692,11 @@ def is_active_bluestore_device(dev):
         return False

     vg_name = lvm.list_lvm_volume_group(dev)
- lv_name = lvm.list_logical_volumes('vg_name={}'.format(vg_name))[0]
+ logical_volumes = lvm.list_logical_volumes('vg_name={}'.format(vg_name))
+ if logical_volumes:
+ lv_name = logical_volumes[0]
+ else:
+ return False

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/659156

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659355
Committed: https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=cd5360736d217cedb80fed6538749508b240b236
Submitter: Zuul
Branch: master

commit cd5360736d217cedb80fed6538749508b240b236
Author: David Ames <email address hidden>
Date: Wed May 15 10:58:18 2019 -0700

    Validate output of list_logical_volumes

    The charm was checking for the zeroth value of the return value of
    list_logical_volumes. However, if no logical volumes are found it
    returns an empty list.

    This change validates that the list has an entry.

    Change-Id: I75a6b1dda15dd7c2cece8cfe97b28317b3d5162b
    Partial-Bug: #1819382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659156
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=e6086328f4e672192377d78c08d652b1aebc0625
Submitter: Zuul
Branch: master

commit e6086328f4e672192377d78c08d652b1aebc0625
Author: David Ames <email address hidden>
Date: Tue May 14 13:24:38 2019 -0700

    Validate output of list_logical_volumes

    The charm was checking for the zeroth value of the return value of
    list_logical_volumes. However, if no logical volumes are found it
    returns an empty list.

    This change validates that the list has an entry.

    Depends-On: I75a6b1dda15dd7c2cece8cfe97b28317b3d5162b
    Change-Id: I2d371dae94dca328cf4782a79e85c1c6fd77f547
    Closes-Bug: #1819382