Provision failed in multipath configuration

[Nastya Urlapova]

Provision failed with error in astute.log:
"Unexpected error\nUnexpected error while running command.\nCommand: pvcreate --metadatacopies 2 --metadatasize 28m /dev/sde3\nExit code: 5\nStdout: ''\nStderr: 'File descriptor 3 (/run/lock/provision.lock) leaked on pvcreate invocation. Parent PID 7122: /usr/bin/python2.7\\nFile descriptor 4 (/var/log/fuel-agent.log) leaked on pvcreate invocation. Parent PID 7122: /usr/bin/python2.7\\n Device /dev/sde3 not found (or ignored by filtering).\\n'\n",
   :exit_code=>255}}

Scenario:
        1. Create cluster with 1 controller, 1 compute and 1 cinder roles
        2. Run network verification
>>>> 3. Provision the cluster <<< failed here
        4. Verify multipath devices on nodes
        5. Deploy the cluster
        6. Run OSTF

Version:
9.2 snapshot #684

[Nastya Urlapova (aurlapova)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.deploy_multipath/157/testReport/(root)/deploy_multipath/deploy_multipath/

[Vladimir Kozhukalov (kozhukalov)]

This issue appears when nailgun-agent starts before device mapper assembles multipath devices and reports them as usual block devices. As a workaround for this issue I suggest to introduce 30 sec delay before starting nailgun-agent first time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/416661

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416661
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=8fbc4d640535387b402aa38cb6227df11c010ff0
Submitter: Jenkins
Branch: stable/mitaka

commit 8fbc4d640535387b402aa38cb6227df11c010ff0
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 4 19:37:10 2017 +0300

    Fix multipath device search

    When nailgun-agent starts before device mapper
    assembles multipath devices it reports physical disks
    that are used for multipath devices as usual disks.

    This patch does the following

    * filter out devices that are DM_MULTIPATH_DEVICE_PATH = 1
    * run 'udevadm settle' before scanning multipath devices
    * delays scanning multipath devices until building block device info

    Change-Id: I088aede0cf3bd1d16a57e7cdec4e50cab2c19175
    Closes-Bug: #1652788

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/418429

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418430

[Dmitry Belyaninov (dbelyaninov)]

waiting https://bugs.launchpad.net/fuel/+bug/1656872

[Dmitry Belyaninov (dbelyaninov)]

Appropriate 9.X SWARM test passed since #178 run
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.deploy_multipath/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418429
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=6b864564bc1a24782343b365de3bc52f7822520c
Submitter: Jenkins
Branch: stable/newton

commit 6b864564bc1a24782343b365de3bc52f7822520c
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 4 19:37:10 2017 +0300

    Fix multipath device search

    When nailgun-agent starts before device mapper
    assembles multipath devices it reports physical disks
    that are used for multipath devices as usual disks.

    This patch does the following

    * filter out devices that are DM_MULTIPATH_DEVICE_PATH = 1
    * run 'udevadm settle' before scanning multipath devices
    * delays scanning multipath devices until building block device info

    Change-Id: I088aede0cf3bd1d16a57e7cdec4e50cab2c19175
    Closes-Bug: #1652788
    (cherry picked from commit 8fbc4d640535387b402aa38cb6227df11c010ff0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418430
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=fd0f4664b381067d04c7c26f3e634bc51ab145fe
Submitter: Jenkins
Branch: master

commit fd0f4664b381067d04c7c26f3e634bc51ab145fe
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 4 19:37:10 2017 +0300

    Fix multipath device search

    When nailgun-agent starts before device mapper
    assembles multipath devices it reports physical disks
    that are used for multipath devices as usual disks.

    This patch does the following

    * filter out devices that are DM_MULTIPATH_DEVICE_PATH = 1
    * run 'udevadm settle' before scanning multipath devices
    * delays scanning multipath devices until building block device info

    Change-Id: I088aede0cf3bd1d16a57e7cdec4e50cab2c19175
    Closes-Bug: #1652788
    (cherry picked from commit 8fbc4d640535387b402aa38cb6227df11c010ff0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-nailgun-agent 11.0.0.0rc1 release candidate.