hook "swift-storage-relation-joined" failed

Bug #1577408 reported by Francis Ginther
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Server
High
Chris Glass
16.06
High
Chris Glass
swift-storage (Juju Charms Collection)
Undecided
Chris Glass

Bug Description

This appears to be a problem with the swift-storage charm not handling symlinks.

2016-05-02 03:26:18 INFO juju-log swift-storage:70: Valid ensured block devices: [u'/dev/disk/by-id/usb-PNY_USB_3.0_FD_587961698-0:0']
2016-05-02 03:26:18 INFO swift-storage-relation-joined Traceback (most recent call last):
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 202, in <module>
2016-05-02 03:26:18 INFO swift-storage-relation-joined main()
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 194, in main
2016-05-02 03:26:18 INFO swift-storage-relation-joined hooks.execute(sys.argv)
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-05-02 03:26:18 INFO swift-storage-relation-joined self._hooks[hook_name]()
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift-storage-relation-joined", line 128, in swift_storage_relation_joined
2016-05-02 03:26:18 INFO swift-storage-relation-joined remember_devices(devs)
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/lib/swift_storage_utils.py", line 383, in remember_devices
2016-05-02 03:26:18 INFO swift-storage-relation-joined blk_uuid = get_device_blkid("/dev/%s" % (dev))
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/lib/swift_storage_utils.py", line 362, in get_device_blkid
2016-05-02 03:26:18 INFO swift-storage-relation-joined blk_uuid = subprocess.check_output(['blkid', '-s', 'UUID', dev])
2016-05-02 03:26:18 INFO swift-storage-relation-joined File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2016-05-02 03:26:18 INFO swift-storage-relation-joined raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['blkid', '-s', 'UUID', u'/dev/usb-PNY_USB_3.0_FD_587961698-0:0']' returned non-zero exit status 2
2016-05-02 03:26:18 ERROR juju.worker.uniter.operation runhook.go:107 hook "swift-storage-relation-joined" failed: exit status 1

A better formatted out is here:
https://pastebin.canonical.com/155617/

All three swift storage units failed with the same signature.

This was observed when deploying a landscape OSA cloud with the maas-1.9 FF enabled.

Revision history for this message
Francis Ginther (fginther) wrote :
Revision history for this message
Francis Ginther (fginther) wrote :

Logs from swift-storage/1.

Revision history for this message
Francis Ginther (fginther) wrote :

Logs from swift-storage/2.

Revision history for this message
Francis Ginther (fginther) wrote :

I can retest this without maas-1.9 enabled if a baseline comparison is needed.

information type: Proprietary → Public
affects: landscape → swift-storage (Juju Charms Collection)
description: updated
tags: added: kanban-cross-team lanscape
summary: - hook "swift-storage-relation-joined" failed with maas-1.9 enabled
+ hook "swift-storage-relation-joined" failed
tags: added: landscape
removed: lanscape
tags: removed: kanban-cross-team
Revision history for this message
Francis Ginther (fginther) wrote :

I attempted a second deployment and it failed with the same symptoms. Here's what those devices look like on one of the swift-storage units:

ubuntu@hawking:~$ ll /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 100 May 3 01:18 ./
drwxr-xr-x 6 root root 120 May 3 01:18 ../
lrwxrwxrwx 1 root root 9 May 3 01:19 ata-BP4_mSATA_SSD_3AF60742166700005324 -> ../../sda
lrwxrwxrwx 1 root root 10 May 3 01:19 ata-BP4_mSATA_SSD_3AF60742166700005324-part1 -> ../../sda1
lrwxrwxrwx 1 root root 9 May 3 01:24 usb-General_USB_Flash_Disk_0180000000014684-0:0 -> ../../sdb

Chris Glass (tribaal)
Changed in swift-storage (Juju Charms Collection):
assignee: nobody → Chris Glass (tribaal)
status: New → In Progress
James Page (james-page)
Changed in swift-storage (Juju Charms Collection):
milestone: none → 16.07
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-swift-storage (master)

Fix proposed to branch: master
Review: https://review.openstack.org/313263

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-swift-storage (master)

Change abandoned by Chris Glass (<email address hidden>) on branch: master
Review: https://review.openstack.org/313263
Reason: Seems like despite my local testing this fix does not properly pass CI. I will resubmit a fix after further testing.

Revision history for this message
Chris Glass (tribaal) wrote :

I dug in and tried a fix for this, but the proposed branch did not actually fix the bug.

After more debugging, I think the problem is the following:

The charm likes to use os.path.baseame(device) and reconstructs device paths with os.path.join("/dev", basename). We pass devices paths as /dev/disks/by-id entries, so given the basename usage, the deployment succeeds only if by chance the basename of what we pass is the same as the actual /dev/* entry.

The new maas code passes the human-friendly name (/dev/disk/by-id/usb-General_USB_Flash_Disk_0180000000014684-0:0 in the given example), but of course, that is a symlink to ../../sdb . Since "/dev/usb-General_USB_Flash_Disk_0180000000014684-0:0" does not exist (notice the absence of /dev/disk/by-id/), the code fails.

I will submit another attempt at fixing this as soon as possible.

Revision history for this message
Chris Glass (tribaal) wrote :

Submitted for review in openstack's codebase again:

Fix proposed to branch: master
Review: https://review.openstack.org/313263

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Added landscape task so that we update the charm revision once it's in the stable charm.

Changed in landscape:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-swift-storage (master)

Reviewed: https://review.openstack.org/313263
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=30c3fb9353d69944f815eb1be6a2c8b1e858fe2b
Submitter: Jenkins
Branch: master

commit 30c3fb9353d69944f815eb1be6a2c8b1e858fe2b
Author: Chris Glass <email address hidden>
Date: Fri May 6 06:51:50 2016 +0000

    Resolve links before using path as block device

    If the charm code is passed symlinks to block devices (as is often the
    case with newer MAAS substrate versions), resolve links before
    attempting to use the block device for storage.

    Charmhelpers were updated as well.

    Testing done:

    - Unit tests pass
    - Tests pass
    - Multiple Openstack Autopilot deployments pass

    Change-Id: If966239502d0752c86e46f3f0aee96f43828aa08
    Closes-Bug: 1577408
    Signed-off-by: Chris Glass <email address hidden>

Changed in swift-storage (Juju Charms Collection):
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-swift-storage (stable/16.04)

Fix proposed to branch: stable/16.04
Review: https://review.openstack.org/314557

Changed in landscape:
milestone: none → 16.06
status: New → Fix Committed
Changed in landscape:
assignee: nobody → Chris Glass (tribaal)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-swift-storage (stable/16.04)

Reviewed: https://review.openstack.org/314557
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=2e1f2f993798e1a50997a1579bbbe9c061f00fee
Submitter: Jenkins
Branch: stable/16.04

commit 2e1f2f993798e1a50997a1579bbbe9c061f00fee
Author: Chris Glass <email address hidden>
Date: Fri May 6 06:51:50 2016 +0000

    Resolve links before using path as block device

    If the charm code is passed symlinks to block devices (as is often the
    case with newer MAAS substrate versions), resolve links before
    attempting to use the block device for storage.

    Charmhelpers were updated as well.

    Testing done:

    - Unit tests pass
    - Tests pass
    - Multiple Openstack Autopilot deployments pass

    Change-Id: If966239502d0752c86e46f3f0aee96f43828aa08
    Closes-Bug: 1577408
    Signed-off-by: Chris Glass <email address hidden>

Liam Young (gnuoy)
Changed in swift-storage (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers