os_bricks fails to rerty "iscsiadm -m session" when iser_use_multipath and iscsi_use_multipath are set in nova.conf

[Mat Clarke]

When attempting to live migrate and VM to a new compute node it fails and remains on the original compute node.

Reviewing the logs I can see the iscsi session is currently not connected and needs to be brought up but it then fails with "ERROR oslo_messaging.rpc.server TargetPortalNotFound: Unable to find target portal 1.1.1.1:3260".

With "iser_use_multipath" and "iscsi_use_multipath" set false in nova.conf I can see the initial os_bricks fails with "No Active sessions" as iscsid has yet to bring up the session but the second try from os_bricks then succeeds as by this time iscsid has brought the session up.

With "iser_use_multipath" and "iscsi_use_multipath" set true in nova.conf I can see the initial os_bricks fails with "No Active sessions" as iscsid has yet to bring up the session but no second attempt from os_bricks leads to the "TargetPortalNotFound".

I'm running "os_brick-2.5.10". should os_bricks retry when using multipath in nova.conf?

[Sophie Huang (sh879n)]

I have recently reported an os-brick bug https://bugs.launchpad.net/os-brick/+bug/1944474 and pushed a patchset for review: https://review.opendev.org/c/openstack/os-brick/+/810331
That patchset might fix your issue. You can try it out.

[Mat Clarke (matclarke-clarkesnet)]

We are currently on Openstack rocky but are planning to upgrade to train
in the next few weeks once upgraded I'll try the patch

On 24/09/2021 15:35, Sophie Huang wrote:
> I have recently reported an os-brick bug https://bugs.launchpad.net/os-brick/+bug/1944474 and pushed a patchset for review: https://review.opendev.org/c/openstack/os-brick/+/810331
> That patchset might fix your issue. You can try it out.
>

[Mat Clarke (matclarke-clarkesnet)]

Hi Sophie, I've back ported your patch to Openstack Rocky but I can't
see any retries. Just a single try on each of the compute nodes we have
in our test environment. I've attached a exert form the logs showing
nova failed to find that volumes and iscsid logs.

We are using iSCSI iSER rather than just iSCSI. Could that be having an
effect?

Thanks
Mat

On 24/09/2021 16:24, Mat Clarke (GMail) wrote:
> We are currently on Openstack rocky but are planning to upgrade to
> train in the next few weeks once upgraded I'll try the patch
>
> On 24/09/2021 15:35, Sophie Huang wrote:
>> I have recently reported an os-brick bug
>> https://bugs.launchpad.net/os-brick/+bug/1944474 and pushed a
>> patchset for review:
>> https://review.opendev.org/c/openstack/os-brick/+/810331
>> That patchset might fix your issue. You can try it out.
>>

[Sofia Enriquez (lsofia-enriquez)]

Fix proposed https://review.opendev.org/c/openstack/os-brick/+/810331

[Sophie Huang (sh879n)]

Mat,

If you are using Rocky, can you verify if you have this PS: https://review.opendev.org/c/openstack/os-brick/+/775545. That is a fix for this bug: https://bugs.launchpad.net/os-brick/+bug/1915678. I don't have iSCSI iSER configured...

[Sophie Huang (sh879n)]

Mat,

Do you know the exit code of your failed command? The retry I added only handles exit code 6...

Sophie

[Mat Clarke (matclarke-clarkesnet)]

Sorry I didn't capture the exit code. I'm currently half way through the
upgrade from Rocky to Train. I'll test once I'm fully migrated.

On 04/10/2021 16:50, Sophie Huang wrote:
> Mat,
>
> Do you know the exit code of your failed command? The retry I added only
> handles exit code 6...
>
> Sophie
>

[Mat Clarke (matclarke-clarkesnet)]

I've now completed upgrade from rocky to train and the iscsiadm is still
failing on the first try but is retrying and succeeding on the second
try without your patch.

*2021-10-08 12:27:08.404* e3-211-rccn01 nova-compute.service 6
2021-10-08 12:27:08.402 4525 INFO os_brick.initiator.connectors.iscsi
[req-38f33f8a-4f80-4a75-a863-11be44ea4f79
630092b76ad44c22ac1d9cbfef23ad88 83f353251c8c4ddb8a70cd1a3c393d34 -
default default] Trying to connect to iSCSI portal 172.20.213.200:3260
*2021-10-08 12:27:05.214* e3-211-rccn01 nova-compute.service 4
2021-10-08 12:27:05.212 4525 WARNING os_brick.initiator.connectors.iscsi
[req-38f33f8a-4f80-4a75-a863-11be44ea4f79
630092b76ad44c22ac1d9cbfef23ad88 83f353251c8c4ddb8a70cd1a3c393d34 -
default default] iscsiadm stderr output when getting sessions: iscsiadm:
No active sessions.
*2021-10-08 12:27:05.139* e3-211-rccn01 nova-compute.service 6
2021-10-08 12:27:05.138 4525 INFO os_brick.initiator.connectors.iscsi
[req-38f33f8a-4f80-4a75-a863-11be44ea4f79
630092b76ad44c22ac1d9cbfef23ad88 83f353251c8c4ddb8a70cd1a3c393d34 -
default default] Trying to connect to iSCSI portal 172.20.213.200:3260

Thanks
Mat

On 05/10/2021 09:26, Mat Clarke (GMail) wrote:

> Sorry I didn't capture the exit code. I'm currently half way through
> the upgrade from Rocky to Train. I'll test once I'm fully migrated.
>
> On 04/10/2021 16:50, Sophie Huang wrote:
>> Mat,
>>
>> Do you know the exit code of your failed command? The retry I added only
>> handles exit code 6...
>>
>> Sophie
>>