JobRequestFailed discovering iSCSI initiator

[Eric Berglund]

Discovering the the iscsi initiator in get_volume_connector [1] can fail with a JobRequestFailed exception if using an older VIOS that does not have the iscsiadm command built in. This case should be handled when discovering the initiator [2].

[1] https://github.com/openstack/nova-powervm/blob/af31c9f97/nova_powervm/virt/powervm/driver.py#L1106
[2] https://github.com/openstack/nova-powervm/blob/af31c9f97/nova_powervm/virt/powervm/volume/iscsi.py#L73-L80

------
TRACE
------
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR pypowervm.wrappers.job [None req-5bd87f90-f333-46cb-ae39-bbad27602457 admin admin] The 'ISCSIDiscovery' operation failed. com.ibm.pmc.jaxb.api.server.jobs.PmcJobException: Error occured while querying the initiator name for the AIX VIOS
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: Not a valid command: iscsiadm -initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: at com.ibm.pmc.rest.servlet.uom.phyp.internal.operations.uom.VirtualIOServerISCSIDiscovery.populateISCSIInitiatorNameAIX(VirtualIOServerISCSIDiscovery.java:315)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: at com.ibm.pmc.rest.servlet.uom.phyp.internal.operations.uom.VirtualIOServerISCSIDiscovery.execute(VirtualIOServerISCSIDiscovery.java:209)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: at com.ibm.pmc.jaxb.api.server.jobs.AbstractPmcJob.execute(AbstractPmcJob.java:729)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR pypowervm.wrappers.job [None req-5bd87f90-f333-46cb-ae39-bbad27602457 admin admin] The 'ISCSIDiscovery' operation failed. Error occured while querying the initiator name for the AIX VIOS
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: Not a valid command: iscsiadm -initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [None req-5bd87f90-f333-46cb-ae39-bbad27602457 admin admin] [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] Failed to attach f98eae11-223b-4f2d-8d9e-dd34d029b43e at /dev/sdb: JobRequestFailed: The 'ISCSIDiscovery' operation failed. Error occured while querying the initiator name for the AIX VIOS
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: Not a valid command: iscsiadm -initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] Traceback (most recent call last):
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova/nova/compute/manager.py", line 5339, in _attach_volume
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] do_driver_attach=True)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] ret_val = method(obj, context, *args, **kwargs)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova/nova/virt/block_device.py", line 634, in attach
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] virt_driver, do_driver_attach)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] return f(*args, **kwargs)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova/nova/virt/block_device.py", line 631, in _do_locked_attach
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] self._do_attach(*args, **_kwargs)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova/nova/virt/block_device.py", line 607, in _do_attach
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] connector = virt_driver.get_volume_connector(instance)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/driver.py", line 1106, in get_volume_connector
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] initiator_dict = iscsi.get_iscsi_initiators(self.adapter)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/volume/iscsi.py", line 89, in get_iscsi_initiators
May 10 11:39:4
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/volume/iscsi.py", line 89, in get_iscsi_initiators
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] discover_initiator(vios_id)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/volume/iscsi.py", line 82, in discover_initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] _discover_initiator()
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] return f(*args, **kwargs)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/volume/iscsi.py", line 75, in _discover_initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] adapter, vios_id)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/usr/local/lib/python2.7/dist-packages/pypowervm/tasks/hdisk/_iscsi.py", line 192, in discover_iscsi_initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] job_wrapper.run_job(vios_uuid, timeout=120)
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] File "/usr/local/lib/python2.7/dist-packages/pypowervm/wrappers/job.py", line 251, in run_job
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] raise exc
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] JobRequestFailed: The 'ISCSIDiscovery' operation failed. Error occured while querying the initiator name for the AIX VIOS
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b]
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b] Not a valid command: iscsiadm -initiator
May 10 11:39:45 esberglu-vol-attach nova-compute[8115]: ERROR nova.compute.manager [instance: 0978e5d4-8065-4594-ab1d-78aa1364579b]

[Eric Berglund (esberglu)]

http://paste.openstack.org/show/720780/

Paste link for better trace formatting

[Matthew Edmonds (edmondsw)]

fix: https://review.openstack.org/#/c/567656/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567656
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=ef77fbfa8c3d8a7feac6f33615e23748056869c0
Submitter: Zuul
Branch: master

commit ef77fbfa8c3d8a7feac6f33615e23748056869c0
Author: Matthew Edmonds <email address hidden>
Date: Thu May 10 13:38:52 2018 -0400

    Handle errors getting iSCSI initiators

    pypowervm may raise a JobRequestFailed exception during iSCSI
    initiator discovery on older VIOS that do not have the iscsiadm
    commands built in. This adds handling for that exception so that
    get_volume_connector doesn't blow up.

    Closes-Bug: #1770466
    Change-Id: I16beff6a5014f028181a167ecb28952190fac223

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova-powervm 7.0.0.0b2 development milestone.