ValueError for Invalid InitiatorConnector in s390

[Arne Recknagel]

Description
===========
Calling the InitiatorConnector factory results in a ValueError for unsupported protocols, which goes unhandled and may crash a calling service.

Steps to reproduce
==================
- clone devstack
- make stack

Expected result
===============
The nova compute service should run.

Actual result
=============
A ValueError is thrown, which, in the case of the nova libvirt driver, is not handled appropriately. The compute service crashes.

Environment
===========
os|distro=kvmibm1
os|vendor=kvmibm
os|release=1.1.3-beta4.3
git|cinder|master[f6ab36d]
git|devstack|master[928b3cd]
git|nova|master[56138aa]
pip|os-brick|1.7.0

Logs & Configs
==============
[...]
2016-11-03 17:56:57.204 46141 INFO nova.virt.driver [req-fb30a5af-e87c-4ee0-903c-a5aa7d3ad5e3 - -] Loading compute driver 'libvirt.LibvirtDriver'
2016-11-03 17:56:57.442 46141 DEBUG os_brick.initiator.connector [req-fb30a5af-e87c-4ee0-903c-a5aa7d3ad5e3 - -] Factory for ISCSI on s390x factory /usr/lib/python2.7/site-packages/os_brick/initiator/connector.py:261
2016-11-03 17:56:57.444 46141 DEBUG os_brick.initiator.connector [req-fb30a5af-e87c-4ee0-903c-a5aa7d3ad5e3 - -] Factory for ISCSI on s390x factory /usr/lib/python2.7/site-packages/os_brick/initiator/connector.py:261
2016-11-03 17:56:57.445 46141 DEBUG os_brick.initiator.connector [req-fb30a5af-e87c-4ee0-903c-a5aa7d3ad5e3 - -] Factory for ISER on s390x factory /usr/lib/python2.7/site-packages/os_brick/initiator/connector.py:261
2016-11-03 17:56:57.445 46141 CRITICAL nova [req-fb30a5af-e87c-4ee0-903c-a5aa7d3ad5e3 - -] ValueError: Invalid InitiatorConnector protocol specified ISER
2016-11-03 17:56:57.445 46141 ERROR nova Traceback (most recent call last):
2016-11-03 17:56:57.445 46141 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2016-11-03 17:56:57.445 46141 ERROR nova sys.exit(main())
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/cmd/compute.py", line 56, in main
2016-11-03 17:56:57.445 46141 ERROR nova topic=CONF.compute_topic)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/service.py", line 216, in create
2016-11-03 17:56:57.445 46141 ERROR nova periodic_interval_max=periodic_interval_max)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/service.py", line 91, in __init__
2016-11-03 17:56:57.445 46141 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/compute/manager.py", line 537, in __init__
2016-11-03 17:56:57.445 46141 ERROR nova self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/virt/driver.py", line 1625, in load_compute_driver
2016-11-03 17:56:57.445 46141 ERROR nova virtapi)
2016-11-03 17:56:57.445 46141 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2016-11-03 17:56:57.445 46141 ERROR nova return import_class(import_str)(*args, **kwargs)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 356, in __init__
2016-11-03 17:56:57.445 46141 ERROR nova self._get_volume_drivers(), self._host)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/virt/driver.py", line 44, in driver_dict_from_config
2016-11-03 17:56:57.445 46141 ERROR nova driver_registry[driver_type] = driver_class(*args, **kwargs)
2016-11-03 17:56:57.445 46141 ERROR nova File "/opt/stack/nova/nova/virt/libvirt/volume/iser.py", line 34, in __init__
2016-11-03 17:56:57.445 46141 ERROR nova transport=self._get_transport())
2016-11-03 17:56:57.445 46141 ERROR nova File "/usr/lib/python2.7/site-packages/os_brick/initiator/connector.py", line 285, in factory
2016-11-03 17:56:57.445 46141 ERROR nova raise ValueError(msg)
2016-11-03 17:56:57.445 46141 ERROR nova ValueError: Invalid InitiatorConnector protocol specified ISER
2016-11-03 17:56:57.445 46141 ERROR nova

[Matt Riedemann (mriedem)]

That's probably b/c there is no mapping for ISER for s390x here:

http://git.openstack.org/cgit/openstack/os-brick/tree/os_brick/initiator/connector.py?h=1.7.0#n149

[Matt Riedemann (mriedem)]

It seems pretty terrible that nova fails to start...is ISER supported on s390x?

[Matt Riedemann (mriedem)]

It would also be much better if os-brick actually raised something besides ValueError for an invalid connector mapping so that nova could handle that and ignore it.

[Matt Riedemann (mriedem)]

os-brick change: https://review.openstack.org/#/c/393941/

[Arne Recknagel (arecknag)]

Nova could catch that exception in this line: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/driver.py#n44

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/394425

[Arne Recknagel (arecknag)]

I submitted a patch for nova that addresses the issue on their side: https://review.openstack.org/#/c/394425/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393941
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=f7b8d242f7125242eebf0f72532daa931d5bb59f
Submitter: Jenkins
Branch: master

commit f7b8d242f7125242eebf0f72532daa931d5bb59f
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 4 18:07:06 2016 -0400

    Raise specific exception for an invalid protocol connector

    Nova's libvirt driver attempts to load up a set of volume
    drivers which use os-brick connectors on start of the
    nova-compute service. If there is a missing or invalid
    mapping, like ISER + s390x, it results in a ValueError and
    nova-compute fails to start.

    This change raises a specific exception, which is still a
    ValueError for backward compatibility, so that Nova can
    handle this situation gracefully and not crash.

    Change-Id: I1d016c226416cf2b77d6e0393498905f56fd8cde
    Partial-Bug: #1639239

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/395067

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/394425
Reason: Abandoning this in favor of https://review.openstack.org/#/c/395067/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/395067
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=57f46c1c28a2e3ba1f12ec8f883cf99ccd9979db
Submitter: Jenkins
Branch: master

commit 57f46c1c28a2e3ba1f12ec8f883cf99ccd9979db
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 8 11:22:42 2016 -0500

    libvirt: handle os-brick InvalidConnectorProtocol on init

    On init the LibvirtDriver imports it's volume drivers into
    a registry for later access during volume operations. Several
    of the libvirt volume drivers are using os-brick for the connector.
    os-brick can raise an exception when loading a connector class
    under certain conditions, e.g. s390x + ISER is not a supported
    mapping in os-brick. This in turn causes the LibvirtDriver to
    fail to load which causes nova-compute to crash.

    This change handles the os-brick specific InvalidConnectorProtocol
    exception when importing volume drivers, logs a debug message but
    doesn't fail to init the LibvirtDriver.

    As part of this change the driver_dict_from_config helper method
    is integrated with the _get_volume_drivers method as that was
    the only thing using it, and it's cleaner if those are just a
    single method.

    Depends-On: I7183a479bc12c4e5509f52bf7fdd0fa7ccaa7247

    Change-Id: I61376469a409dc5475886e6c79486b815129392a
    Closes-Bug: #1639239

[Ryan Beisner (1chb1n)]

To confirm, we are seeing this with s390x architectures running Newton:

2016-11-30 20:55:31.178 45484 CRITICAL nova [req-226c429a-9a55-4e85-b8e8-623bc32f3509 - - - - -] ValueError: Invalid InitiatorConnector protocol specified ISCSI
2016-11-30 20:55:31.178 45484 ERROR nova Traceback (most recent call last):
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2016-11-30 20:55:31.178 45484 ERROR nova sys.exit(main())
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/compute.py", line 60, in main
2016-11-30 20:55:31.178 45484 ERROR nova db_allowed=CONF.conductor.use_local)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 218, in create
2016-11-30 20:55:31.178 45484 ERROR nova db_allowed=db_allowed)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 92, in __init__
2016-11-30 20:55:31.178 45484 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 536, in __init__
2016-11-30 20:55:31.178 45484 ERROR nova self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/virt/driver.py", line 1619, in load_compute_driver
2016-11-30 20:55:31.178 45484 ERROR nova virtapi)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-11-30 20:55:31.178 45484 ERROR nova return import_class(import_str)(*args, **kwargs)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 354, in __init__
2016-11-30 20:55:31.178 45484 ERROR nova self._get_volume_drivers(), self)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/virt/driver.py", line 44, in driver_dict_from_config
2016-11-30 20:55:31.178 45484 ERROR nova driver_registry[driver_type] = driver_class(*args, **kwargs)
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 41, in __init__
2016-11-30 20:55:31.178 45484 ERROR nova transport=self._get_transport())
2016-11-30 20:55:31.178 45484 ERROR nova File "/usr/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 273, in factory
2016-11-30 20:55:31.178 45484 ERROR nova raise ValueError(msg)
2016-11-30 20:55:31.178 45484 ERROR nova ValueError: Invalid InitiatorConnector protocol specified ISCSI

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428364
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=e1b77f95252a377151908f58ad7db62e27f7217c
Submitter: Jenkins
Branch: stable/newton

commit e1b77f95252a377151908f58ad7db62e27f7217c
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 4 18:07:06 2016 -0400

    Raise specific exception for an invalid protocol connector

    Nova's libvirt driver attempts to load up a set of volume
    drivers which use os-brick connectors on start of the
    nova-compute service. If there is a missing or invalid
    mapping, like ISER + s390x, it results in a ValueError and
    nova-compute fails to start.

    This change raises a specific exception, which is still a
    ValueError for backward compatibility, so that Nova can
    handle this situation gracefully and not crash.

    Change-Id: I1d016c226416cf2b77d6e0393498905f56fd8cde
    Partial-Bug: #1639239
    (cherry picked from commit f7b8d242f7125242eebf0f72532daa931d5bb59f)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/428386

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436868
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ec3f15994476cf36e0ab34c0d3145102e61fece9
Submitter: Jenkins
Branch: stable/newton

commit ec3f15994476cf36e0ab34c0d3145102e61fece9
Author: Arne Recknagel <email address hidden>
Date: Tue Feb 21 16:56:53 2017 +0100

    Fix s390 "connector not found" issue

    This patch introduces a try-catch block on a call to the os-brick
    library which may throw a ValueError. os_brick will throw it when
    a driver is not supported, so ignoring it and skipping its addition
    to the driver list should be the expected behavior. This fix is
    required for s390, where nova is currently broken without it.

    The issue was resolved in Ocata with changes including [1] and [2],
    as well as a change to os_brick [3]. The backport to newton [4] was
    abandoned as it requires a raise in the os_brick minimum version.
    This fix is a revision of [5], which is essentially a lightweight
    version of [1] that only addresses the issue at hand.

    [1]: https://review.openstack.org/#/c/395067/
    [2]: https://review.openstack.org/#/c/394979/
    [3]: https://review.openstack.org/#/c/375415/
    [4]: https://review.openstack.org/#/c/428386/
    [5]: https://review.openstack.org/#/c/394425/

    Change-Id: I638f12ce3deab0338a8d2652398c35acb9280f72
    Closes-Bug: 1639239

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.5 release.

[Andreas Scheuring (andreas-scheuring)]

Can anybody help me clarifying the state in the ubuntu packages for this?

I'm running a Xenial with Newton OpenStack on s390x. I have the following nova-common package installed: 2:13.0.0-0ubuntu2.

I must admit I don't understand all the projects that are listed above. It seems like the fix got committed for the newton cloud archive but not yet released. Is that true? I also don't get why there are so many projects (ubuntu cloud archive, openstack compute (nova), ...)

If somebody could clarify the state, that would be very helpful!

[Dimitri John Ledkov (xnox)]

There are so many projects in launchpad for this bug report, because it is not just the Ubuntu Project bug tracker, but also a bug tracker for OpenStack upstream as well.

Multiple Openstack releases can be used on Ubuntu. With Xenial we offer Mitaka, Newton and Ocata. Mitaka is what Xenial shipped with originally, Newton & Ocata are only available from the optional Cloud Archive repository https://wiki.ubuntu.com/OpenStack/CloudArchive

Based on the package version number, it seems to me that Mitaka release is installed. And based on the bug statuses it does not appear that this bug is fixed in Xenial/Mitaka.

[Marco Pavone (pavone)]

Regarding newton: the nova patch has landed in 14.0.5
The bug entry for nova source of stable/newton was in the wrong state. It is now set manually to Released.

However, for the newton ubuntu cloud archive I'm not sure: there is a commit in the stable/newton ubuntu repo that tagged the nova version 14.0.5 (https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/nova/commit/?h=stable/newton&id=837d87810d483bcb0947b6c4b389c9344297d3ce) - but the deb package is not (yet?) available in the cloud archive.

[Corey Bryant (corey.bryant)]

Marking this as fix-committed for yakkety/newton since this is in the 14.0.5 release which is in yakkety-proposed and newton-proposed at the moment.

[Corey Bryant (corey.bryant)]

My previous comment was for nova only. I've uploaded a new version of python-os-brick that includes this fix to the yakkety unapproved queue for SRU team review.