Querying Windows via WMI intermittently fails in get_device_number_for_target

[Jay Bryant]

The WMI query in basevolumeutils.get_device_number_for_target can incorrectly return no device_number during extended stress runs. We saw that it would return incorrect data after about an hour. It could also return an initiator_sessions object that was empty.

By adding a check to make sure that devices wasn't empty and adding a retry loop in volumeops._get_mounted_disk_from_lun we could avoid hitting the case where it thought it couldn't get a mounted disk for the target_iqn.

[Alessandro Pilotti (alexpilotti)]

Hi Jay,

Good catch, can you provide some info on your environment and possibly the integration tests that you are using to duplicate the issue?

Can you please post a pastebin with the Nova compute log as well?

Tx

[Jay Bryant (jsbryant)]

Alex,

The environment we were running in was using iSCSI from a Linux based control node. The volumes were on LVM.

We started seeing the attachments fail intermittently, after about an hour, with:
raise exception.NotFound(_('Unable to find a mounted disk for '
                                                     'target_iqn: %s') % target_iqn)

The person who has access to the system where this is happening is out on vacation right now. So, I can't get you a pastebin right now. I have a patch written up. We ran with the patch for 7 hours without seeing the problem recreate.

Is this type of patch the sort of thing you expect to have a unit test as well?

[Jay Bryant (jsbryant)]

As far as how we encountered the problem, I believe it was while running tempest.

[Matt Riedemann (mriedem)]

Patch: https://review.openstack.org/#/c/55449/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/55449
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d143540ad1b69ec93c2b7bfadd1f654c4d8c7a34
Submitter: Jenkins
Branch: master

commit d143540ad1b69ec93c2b7bfadd1f654c4d8c7a34
Author: Jay S. Bryant <email address hidden>
Date: Wed Nov 6 10:49:00 2013 -0600

    hyperv: Retry after WMI query fails to find dev

    During long stress runs the WMI query that is looking for
    the iSCSI device number can incorrectly return no data.
    If the query is retried the appropriate data can then be
    obtained.

    This commit adds a retry loop, calling
    basevolumeutils.get_device_number_for_target to avoid this situation.
    It also handles the case where the devices list returned in
    get_device_number_for_target is empty. The retry loop is
    implemented with new mounted_disk_query_retry_count and
    mounted_disk_query_retry_interval configuration options.

    Unit tests have been added to check the good and bad paths for
    get_mounted_disk_from_lun.

    DocImpact
    Closes-bug: 1247901
    Change-Id: I082c4b1694efcd20cce65293cd330b7a0cf7d470

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/72771

[Alessandro Pilotti (alexpilotti)]

This fix introduced an issue in the test case:

https://bugs.launchpad.net/nova/+bug/1280379

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72771
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c7ff2af939e4be049f65d87ef17547ee95285a0
Submitter: Jenkins
Branch: stable/havana

commit 1c7ff2af939e4be049f65d87ef17547ee95285a0
Author: Jay S. Bryant <email address hidden>
Date: Wed Nov 6 10:49:00 2013 -0600

    hyperv: Retry after WMI query fails to find dev

    During long stress runs the WMI query that is looking for
    the iSCSI device number can incorrectly return no data.
    If the query is retried the appropriate data can then be
    obtained.

    This commit adds a retry loop, calling
    basevolumeutils.get_device_number_for_target to avoid this situation.
    It also handles the case where the devices list returned in
    get_device_number_for_target is empty. The retry loop is
    implemented with new mounted_disk_query_retry_count and
    mounted_disk_query_retry_interval configuration options.

    Unit tests have been added to check the good and bad paths for
    get_mounted_disk_from_lun.

    DocImpact
    Closes-bug: 1247901
    Change-Id: I082c4b1694efcd20cce65293cd330b7a0cf7d470
    (cherry picked from commit d143540ad1b69ec93c2b7bfadd1f654c4d8c7a34)