[OVN]IPv6 tempest tests are failing with OVN backend

[Maciej Jozefczyk]

Recently merged code [1] added a few IPv6 hotplug scenarios.
In meantime we're working on enabling new Cirros on OVN Gates [2]

After merging [1] we can find that on [2] the new tests started to fail:

neutron_tempest_plugin.scenario.test_ipv6.IPv6Test.test_ipv6_hotplug_dhcpv6stateless
neutron_tempest_plugin.scenario.test_ipv6.IPv6Test.test_ipv6_hotplug_slaac.

Example failure:
https://ef5d43af22af7b1c1050-17fc8f83c20e6521d7d8a3ccd8bca531.ssl.cf2.rackcdn.com/711425/10/check/neutron-ovn-tempest-ovs-release

[1] https://review.opendev.org/#/c/711931/
[2] https://review.opendev.org/#/c/711425/

[Maciej Jozefczyk (maciejjozefczyk)]

The same tests are failing now in stable/braches:

https://review.opendev.org/#/c/731477/

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d9/731477/4/check/networking-ovn-tempest-dsvm-ovs-release/7d9fade/testr_results.html

[Radosław Piliszek (yoctozepto)]

It fails elsewhere too: https://review.opendev.org/732081

[Lajos Katona (lajos-katona)]

Yes, it seems to be quite frequent:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RuntimeError%3A%20Timed%20out%20waiting%20for%20IP%20address%5C%22

[Brian Haley (brian-haley)]

I think the zuul bot will update this, but I added this failing test class to the blacklist in https://review.opendev.org/#/c/732463/

[Slawek Kaplonski (slaweq)]

I don't think it's related to the cirros 0.5.1.
It's just this test was merged just few days ago and it wasn't tested with ovn backend as we don't have any ovn related job in the neutron-tempest-plugin repo.

[Maciej Jozefczyk (maciejjozefczyk)]

Yes, that is not related to Cirros. That is related to Core-OVN.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/733061

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/732463
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1a3811448d0a108e8e16bb32810e4f59db226a4b
Submitter: Zuul
Branch: master

commit 1a3811448d0a108e8e16bb32810e4f59db226a4b
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 18:01:21 2020 -0400

    Bump oslo.log to version 4.1.2

    Versions of oslo.log >= 4.1.2 require a second argument when
    initializing the OSJournalHandler class. While I've started
    a review to fix that regression [0], bump oslo.log to a later
    version that requires this new flag and pass it to fix the
    gate.

    Also had to blacklist the OVN tempest IPv6 hotplug test
    since it is failing too much to pass the check jobs.

    [0] https://review.opendev.org/732457

    Change-Id: Ic9bbb43aa832ad6cc45d57328b40afe4468ddfca
    Related-bug: #1871840
    Related-bug: #1881558

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Józefczyk (<email address hidden>) on branch: master
Review: https://review.opendev.org/733061

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/733785

[Jakub Libosvar (libosvar)]

I just reproduced it on my environment. It seems OVN is not sending out Router Advertisements or it's not delivered to the instance port

[Maciej Jozefczyk (maciejjozefczyk)]

Hey

I noticed that port created during this test having IPv6 HAS NO dhcpv6 options set:

stack@mjozefcz-tempest-plugin-failed-install:~/neutron/neutron$ openstack subnet show 307d0cc8-481f-4052-b933-e4eaa84c1e53
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools | 2001:db8::2-2001:db8::ffff:ffff:ffff:ffff |
| cidr | 2001:db8::/64 |
| created_at | 2020-06-08T13:45:55Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | False |
| enable_dhcp | True |
| gateway_ip | 2001:db8::1 |
| host_routes | |
| id | 307d0cc8-481f-4052-b933-e4eaa84c1e53 |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| location | cloud='', project.domain_id=, project.domain_name=, project.id='97ab32e6e26143f9938405a6e36180f7', project.name=, region_name='RegionOne', zone= |
| name | ...

[Maciej Jozefczyk (maciejjozefczyk)]

Sorry, wrong PASTE, the LSP not having DHCP options set:
http://paste.openstack.org/show/794463/

[Jakub Libosvar (libosvar)]

This is a bug in core OVN - the ACLs are applied on packets generated by ovn-controller instead of being skipped. The neutron_pg_drop table drops router advertisements. Adding an explicit rule to allow ingress RA makes the test pass.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/733785
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d09b1b40b6bded2490ea980bd9c0d3718600de00
Submitter: Zuul
Branch: stable/ussuri

commit d09b1b40b6bded2490ea980bd9c0d3718600de00
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 18:01:21 2020 -0400

    Blacklist the OVN tempest IPv6 hotplug test

    Blacklist the OVN tempest IPv6 hotplug test since it is
    failing too much to pass the check jobs.

    Partial cherry pick of 1a3811448d0a108e8e16bb32810e4f59db226a4b

    Change-Id: Ic9bbb43aa832ad6cc45d57328b40afe4468ddfca
    Related-bug: #1881558
    (cherry picked from commit 1a3811448d0a108e8e16bb32810e4f59db226a4b)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/735553

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/735553
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/827303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/827303
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/88fc3fb64e9901ec64b141cbcad1e0fc5325fedf
Submitter: "Zuul (22348)"
Branch: master

commit 88fc3fb64e9901ec64b141cbcad1e0fc5325fedf
Author: yatinkarel <email address hidden>
Date: Tue Feb 1 20:06:04 2022 +0530

    Re-enable IPv6Test in OVN scenario job

    The related bug is fixed in ovn with[1], so issue
    should not be see with ovn-20.03.2+ releases. This
    patch re-enables the test by removing it from blacklist,
    and adding it to ussuri ovn job as that is running with
    ovn-20.03.0.

    [1] https://github.com/ovn-org/ovn/commit/53285ae

    Depends-On: https://review.opendev.org/c/openstack/networking-sfc/+/827410/
    Related-Bug: #1881558
    Change-Id: I73c3d6caeaf578fb032750c34e1aaa1ac041da9a