hook "swift-storage-relation-joined" failed

[Francis Ginther]

This appears to be a problem with the swift-storage charm not handling symlinks.

2016-05-02 03:26:18 INFO juju-log swift-storage:70: Valid ensured block devices: [u'/dev/disk/by-id/usb-PNY_USB_3.0_FD_587961698-0:0']
2016-05-02 03:26:18 INFO swift-storage-relation-joined Traceback (most recent call last):
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 202, in <module>
2016-05-02 03:26:18 INFO swift-storage-relation-joined main()
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 194, in main
2016-05-02 03:26:18 INFO swift-storage-relation-joined hooks.execute(sys.argv)
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-05-02 03:26:18 INFO swift-storage-relation-joined self._hooks[hook_name]()
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 128, in swift_storage_relation_joined
2016-05-02 03:26:18 INFO swift-storage-relation-joined remember_devices(devs)
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/lib/swift_storage_utils.py", line 383, in remember_devices
2016-05-02 03:26:18 INFO swift-storage-relation-joined blk_uuid = get_device_blkid("/dev/%s" % (dev))
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/lib/swift_storage_utils.py", line 362, in get_device_blkid
2016-05-02 03:26:18 INFO swift-storage-relation-joined blk_uuid = subprocess.check_output(['blkid', '-s', 'UUID', dev])
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2016-05-02 03:26:18 INFO swift-storage-relation-joined raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['blkid', '-s', 'UUID', u'/dev/usb-PNY_USB_3.0_FD_587961698-0:0']' returned non-zero exit status 2
2016-05-02 03:26:18 ERROR juju.worker.uniter.operation runhook.go:107 hook "swift-storage-relation-joined" failed: exit status 1

A better formatted out is here:
https://pastebin.canonical.com/155617/

All three swift storage units failed with the same signature.

This was observed when deploying a landscape OSA cloud with the maas-1.9 FF enabled.

[Francis Ginther (fginther)]

Logs from swift-storage/1.

[Francis Ginther (fginther)]

Logs from swift-storage/2.

[Francis Ginther (fginther)]

I can retest this without maas-1.9 enabled if a baseline comparison is needed.

[Francis Ginther (fginther)]

I attempted a second deployment and it failed with the same symptoms. Here's what those devices look like on one of the swift-storage units:

ubuntu@hawking:~$ ll /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 100 May 3 01:18 ./
drwxr-xr-x 6 root root 120 May 3 01:18 ../
lrwxrwxrwx 1 root root 9 May 3 01:19 ata-BP4_mSATA_SSD_3AF60742166700005324 -> ../../sda
lrwxrwxrwx 1 root root 10 May 3 01:19 ata-BP4_mSATA_SSD_3AF60742166700005324-part1 -> ../../sda1
lrwxrwxrwx 1 root root 9 May 3 01:24 usb-General_USB_Flash_Disk_0180000000014684-0:0 -> ../../sdb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313263

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Glass (<email address hidden>) on branch: master
Review: https://review.openstack.org/313263
Reason: Seems like despite my local testing this fix does not properly pass CI. I will resubmit a fix after further testing.

[Chris Glass (tribaal)]

I dug in and tried a fix for this, but the proposed branch did not actually fix the bug.

After more debugging, I think the problem is the following:

The charm likes to use os.path.baseame(device) and reconstructs device paths with os.path.join("/dev", basename). We pass devices paths as /dev/disks/by-id entries, so given the basename usage, the deployment succeeds only if by chance the basename of what we pass is the same as the actual /dev/* entry.

The new maas code passes the human-friendly name (/dev/disk/by-id/usb-General_USB_Flash_Disk_0180000000014684-0:0 in the given example), but of course, that is a symlink to ../../sdb . Since "/dev/usb-General_USB_Flash_Disk_0180000000014684-0:0" does not exist (notice the absence of /dev/disk/by-id/), the code fails.

I will submit another attempt at fixing this as soon as possible.

[Chris Glass (tribaal)]

Submitted for review in openstack's codebase again:

Fix proposed to branch: master
Review: https://review.openstack.org/313263

[Andreas Hasenack (ahasenack)]

Added landscape task so that we update the charm revision once it's in the stable charm.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313263
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=30c3fb9353d69944f815eb1be6a2c8b1e858fe2b
Submitter: Jenkins
Branch: master

commit 30c3fb9353d69944f815eb1be6a2c8b1e858fe2b
Author: Chris Glass <email address hidden>
Date: Fri May 6 06:51:50 2016 +0000

    Resolve links before using path as block device

    If the charm code is passed symlinks to block devices (as is often the
    case with newer MAAS substrate versions), resolve links before
    attempting to use the block device for storage.

    Charmhelpers were updated as well.

    Testing done:

    - Unit tests pass
    - Tests pass
    - Multiple Openstack Autopilot deployments pass

    Change-Id: If966239502d0752c86e46f3f0aee96f43828aa08
    Closes-Bug: 1577408
    Signed-off-by: Chris Glass <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/16.04
Review: https://review.openstack.org/314557

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314557
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=2e1f2f993798e1a50997a1579bbbe9c061f00fee
Submitter: Jenkins
Branch: stable/16.04

commit 2e1f2f993798e1a50997a1579bbbe9c061f00fee
Author: Chris Glass <email address hidden>
Date: Fri May 6 06:51:50 2016 +0000

    Resolve links before using path as block device

    If the charm code is passed symlinks to block devices (as is often the
    case with newer MAAS substrate versions), resolve links before
    attempting to use the block device for storage.

    Charmhelpers were updated as well.

    Testing done:

    - Unit tests pass
    - Tests pass
    - Multiple Openstack Autopilot deployments pass

    Change-Id: If966239502d0752c86e46f3f0aee96f43828aa08
    Closes-Bug: 1577408
    Signed-off-by: Chris Glass <email address hidden>