Only one NVMe connection may be functional on Compute node

[Hamdy Khader]

Only one NVMe connection is active and working properly on the same Compute.
When I try to attach second volume to 2nd VM on the same Compute node, the operation fails.

Nova log shows error when executing "nvme list" (exit code non-zero)

Reproduction:
1. Deploy tripleO master on 3 nodes
- 1st host: Controller and NVMf target as cinder backend
- 2nd and 3rd host: Compute nodes
2. Spawn VM-1, create volume and attach volume to VM-1
3. Spawn 2nd VM-2 on the same compute node, create new volume and attach the volume to VM-2.

[Hamdy Khader (hamdyk)]

nova commit:
https://review.openstack.org/#/c/602351/

[Hamdy Khader (hamdyk)]

os-brick commit:
https://review.openstack.org/#/c/602332/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602351
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a1325b4c76ba5ad42a09370d6250338cfb3de533
Submitter: Zuul
Branch: master

commit a1325b4c76ba5ad42a09370d6250338cfb3de533
Author: Hamdy Khader <email address hidden>
Date: Thu Sep 13 16:24:54 2018 +0300

    Set defult value of num_nvme_discover_tries=5

    Discovering newly connected devices in the initiator side can be slow
    when there are old connections, increasing the retries is important
    to discover the new connected device.

    Change-Id: I62a8162bf96d51f7252cfefbffc1a46010a3a612
    Closes-Bug: #1792313

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/608683

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602332
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=fab367cc265593ac7b46feca8cb13da1385a3d03
Submitter: Zuul
Branch: master

commit fab367cc265593ac7b46feca8cb13da1385a3d03
Author: Hamdy Khader <email address hidden>
Date: Thu Sep 13 14:38:11 2018 +0300

    Retry executing command "nvme list" when fail

    Only one NVMe connection can be active on the same Compute node, when
    initiator side has more than one connection then new connection is made,
    results of the new connected device is not instantaneous.

    Fixing this issue requires retry and sleep when executing command
    "nvme list" to retrieve the newly connected device.

    Change-Id: I6b70140be7023770b83603de723d6d2de3ebb747
    Closes-Bug: #1792313

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613774
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=a8215ba452c5d4c5f815cd22f49fa2e20122fb31
Submitter: Zuul
Branch: stable/rocky

commit a8215ba452c5d4c5f815cd22f49fa2e20122fb31
Author: Hamdy Khader <email address hidden>
Date: Thu Sep 13 14:38:11 2018 +0300

    Retry executing command "nvme list" when fail

    Only one NVMe connection can be active on the same Compute node, when
    initiator side has more than one connection then new connection is made,
    results of the new connected device is not instantaneous.

    Fixing this issue requires retry and sleep when executing command
    "nvme list" to retrieve the newly connected device.

    Change-Id: I6b70140be7023770b83603de723d6d2de3ebb747
    Closes-Bug: #1792313
    (cherry picked from commit fab367cc265593ac7b46feca8cb13da1385a3d03)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608683
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3d4e4a4df64910a5d3d95c12e9acd0ddba70843f
Submitter: Zuul
Branch: stable/rocky

commit 3d4e4a4df64910a5d3d95c12e9acd0ddba70843f
Author: Hamdy Khader <email address hidden>
Date: Thu Sep 13 16:24:54 2018 +0300

    Set defult value of num_nvme_discover_tries=5

    Discovering newly connected devices in the initiator side can be slow
    when there are old connections, increasing the retries is important
    to discover the new connected device.

    Change-Id: I62a8162bf96d51f7252cfefbffc1a46010a3a612
    Closes-Bug: #1792313
    (cherry picked from commit a1325b4c76ba5ad42a09370d6250338cfb3de533)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.6.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 2.5.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.