[CI][Devstack] Failing to SSH to test VMs with IPv6

[Carlos Eduardo]

Description
===========
The LVM job for manila is failing constantly. Few tests are having the same issue while trying to SSH into instances.

Steps to reproduce
==================

- Configure tempest
- Configure LVM backend
- Run functional tests

Expected result
===============
The job should be able to ssh into the instances and the tests should be passing.

Actual result
=============
The following tests are failing:

{0} manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFSIPv6.test_mount_share_one_vm
{5} manila_tempest_tests.tests.scenario.test_share_extend.TestShareExtendNFSIPv6.test_create_extend_and_write
{0} manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFSIPv6.test_read_mountable_snapshot
{0} manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFSIPv6.test_read_write_two_vms
{0} manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFSIPv6.test_write_data_to_share_created_from_snapshot
{0} manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFSIPv6.test_write_with_ro_access

This exception is being raised: https://paste.openstack.org/show/808158/

Environment
===========
- Ubuntu 20.04
- Manila using the master branch

Logs & Configs
==============

local.conf example: https://paste.openstack.org/show/808159/
manila.conf example: https://paste.openstack.org/show/808160/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/805319

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/805319
Committed: https://opendev.org/openstack/manila-tempest-plugin/commit/8ccbbebe6d424265ff717197b9a323917518e8c5
Submitter: "Zuul (22348)"
Branch: master

commit 8ccbbebe6d424265ff717197b9a323917518e8c5
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Aug 19 17:01:21 2021 -0700

    [CI] Temporarily disable IPv6 scenario tests

    We're experiencing failures to SSH into test
    VMs over IPv6 IP addresses from private tenant
    networks in the LVM driver job when testing
    main branch code. It's possible this is a regression
    in nova, neutron, neutron dynamic routing or bgp. We'll
    drop this test exclusion when the bug has been
    identified and fixed.

    Change-Id: Ib39cb2bda862758f575262647e068dd9deaf1459
    Partial-Bug: #1940324
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[Vida Haririan (vhariria)]

Additional comments at https://meetings.opendev.org/meetings/manila/2022/manila.2022-08-11-15.00.log.html

[Goutham Pacha Ravi (gouthamr)]

Hi Neutron developers,

This doesn't appear to be an issue in manila; but the Manila CI is affected. Here's how the issue can be reproduced:

local.conf: https://paste.opendev.org/show/bHxkDm4bHKPfpNum4uZ5/
----------

We're disabling "NEUTRON_CREATE_INITIAL_NETWORKS" so that we can do it in the manila devstack plugin as part of BGP setup with neutron-dynamic-routing: https://github.com/openstack/manila/blob/85fb7d3625cee0a6706f7822b7b439485f525906/devstack/plugin.sh#L1074-L1223

Is there any test in any CI job where SSH to a nova instance on a tenant network's IPv6 address is tested?

[Bernard Cafarelli (bcafarel)]

We have ipv6-only jobs like neutron-ovn-tempest-ipv6-only:
https://github.com/openstack/neutron/blob/master/zuul.d/tempest-singlenode.yaml#L532
A recent run example is https://zuul.opendev.org/t/openstack/build/d3ae5228c184475da25286b66e37c07c
It is just inheriting from generic tempest-ipv6-only

local.conf does not look too lengthy, I will ask around in neutron team meeting for ideas here

[Lajos Katona (lajos-katona)]

Hi, do you have an example run with all the logs? As I see this job is quite different as it sets up dynamic-routing with quagga....

[Lajos Katona (lajos-katona)]

We dicsussed this issue on our weekly meeting, and agreed to ask you to try to change the job to use ovs instead on ovn, as ovn and neutron-dynamic-routing are not compatible (see [1])

For zuul/local.conf example see the neutron-tempest-plugin-openvswitch job definition how to disable ovn services (see [2]).

Meeting logs (just for reference): [3]

[1]: https://bugs.launchpad.net/neutron/+bug/1959666
[2]: https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/zuul.d/master_jobs.yaml#L219

[3]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-08-16-14.00.log.html#l-64

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/853367

[Goutham Pacha Ravi (gouthamr)]

Thank you Bernard and Lajos; I'll report back after experimenting the same with OVS as you suggested: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/853367

[Goutham Pacha Ravi (gouthamr)]

Hi Lajos, Bernard,

Reverting the job to use OVS works, and the IPv6 tests are now working again! Thanks for your help identifying the issue and solving it with this workaround for now. I'd be happy to switch these jobs to using OVN again when https://launchpad.net/bugs/1959666 is resolved.

[Bernard Cafarelli (bcafarel)]

Nice, glad to hear it!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/853367
Committed: https://opendev.org/openstack/manila-tempest-plugin/commit/f67893ccccb59151b69719bf44c3a09aae541b11
Submitter: "Zuul (22348)"
Branch: master

commit f67893ccccb59151b69719bf44c3a09aae541b11
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Aug 16 14:07:06 2022 -0700

    Use OVS with IPv6 data path testing jobs

    We run IPv6 data-path tests with LVM and
    Ceph-NFS drivers by setting up bgp to
    advertise tenant network routes to the host.
    Bug #1959666 prevents the use of OVN with
    neutron-dynamic-routing. The underlying neutron
    ML2 mechanism driver doesn't matter to
    manila; so lets use OVS in the meanwhile with
    these test jobs.

    Closes-Bug: #1940324
    Related-Bug: #1959666
    Change-Id: I5b37f31e302ce75650f817a937d8f17ee58887b2
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila-tempest-plugin 1.10.0 release.

[Brian Haley (brian-haley)]

So it looks like the neutron work is tracked in the other bug, can we close this now?