Tempest is failing after new test was added to neutron tempest plugin

[Daniel Alvarez]

After this commit [0] landed in neutron tempest plugin, our tempest jobs are failing.
We need to check whether is a test problem or a legitimate bug in networking-ovn/OVN.

The test exercises the following scenario:

     +-----+
     | VM 1|
     +--+--+
        |
        | Network 1 (tenant network)
        |
      +-+-+
      | R1|
      +-+-+
        |
        | Network 3 (tenant network)
        |
      +-+-+
      | R2|
      +-+-+
        |
        | Network 2 (tenant network)
        |
     +--+--+
     | VM 2|
     +-----+

[0] https://github.com/openstack/neutron-tempest-plugin/commit/b1a3289fe4b98d1270844aa81f60604e64ae50cf

[Daniel Alvarez (dalvarezs)]

I've tried to reproduce locally the scenario that this test is exercising and it works good for me:

$ sudo ovn-nbctl show
switch 9bb2d9c0-831f-4527-ae97-6aec8b477f42 (neutron-448eee0d-f336-45aa-9b23-308235785569) (aka private)
    port 39e36b42-3d09-4e28-9bae-2d302a03481e
        type: localport
        addresses: ["fa:16:3e:01:f3:fa 10.0.0.2 fded:d86e:b73f:0:f816:3eff:fe01:f3fa"]
    port 367ac68a-9732-4c91-a458-2fc221491e2f
        type: router
        router-port: lrp-367ac68a-9732-4c91-a458-2fc221491e2f
    port 55edcb76-e904-4dea-8ddb-cb770c3b7aee
        type: router
        router-port: lrp-55edcb76-e904-4dea-8ddb-cb770c3b7aee
    port 51cebfef-39a7-4996-81e6-dd48a4a81dfd
        addresses: ["fa:16:3e:0c:84:d6 10.0.0.10 fded:d86e:b73f:0:f816:3eff:fe0c:84d6"]
switch 62370e9c-8078-4388-88e4-7bbb621fa574 (neutron-ef71d6ca-7640-45c1-aed3-e2b8cef15c28) (aka public)
    port 59f74080-ff5f-474d-9910-f362122d0ddf
        type: router
        router-port: lrp-59f74080-ff5f-474d-9910-f362122d0ddf
    port provnet-ef71d6ca-7640-45c1-aed3-e2b8cef15c28
        type: localnet
        addresses: ["unknown"]
    port 64c896a9-bea0-4a48-b96b-96464df28f3b
        type: localport
        addresses: ["fa:16:3e:c6:dd:fe"]
switch eebb15e4-4d2c-48ad-b19f-894e2c517c6c (neutron-a934cf7d-50f7-4dde-beb2-78cb180a0acc) (aka network2)
    port 184e0495-03f3-483e-b6fa-b978e9ec8905
        type: router
        router-port: lrp-184e0495-03f3-483e-b6fa-b978e9ec8905
    port 67d4135f-db47-45c8-bf1b-c4b0b7d6051b
        type: localport
        addresses: ["fa:16:3e:2f:75:ab 192.168.10.2"]
    port f451bd39-5d44-41e8-b8a4-1ebcc83147c9
        addresses: ["fa:16:3e:3a:41:fc 192.168.10.4"]
switch 6261c14d-c0da-4d2c-b309-16ce563761e0 (neutron-78286b7a-06b6-48b1-ab40-f87ccb23ac1d) (aka network3)
    port ed598f54-6519-4bed-b30c-483ed67878d2 (aka p32)
        type: router
        router-port: lrp-ed598f54-6519-4bed-b30c-483ed67878d2
    port 9e215eaa-e905-4ce9-af1d-ac5c6d2a3b8a (aka p31)
        addresses: ["fa:16:3e:4f:33:5c 192.168.30.7"]
    port 78bd0e86-44ea-4512-b03e-743bb33d877f
        type: localport
        addresses: ["fa:16:3e:71:52:b0 192.168.30.2"]
    port 91075d39-be43-4102-a12d-019f7c40063e
        type: router
        router-port: lrp-91075d39-be43-4102-a12d-019f7c40063e
router 7d415648-8216-4443-ac39-94c462ad7edb (neutron-8d2d2e00-afde-4052-bf30-b9cc93ce4505) (aka router2)
    port lrp-ed598f54-6519-4bed-b30c-483ed67878d2
        mac: "fa:16:3e:6d:3c:eb"
        networks: ["192.168.30.9/24"]
    port lrp-184e0495-03f3-483e-b6fa-b978e9ec8905
        mac: "fa:16:3e:76:6c:62"
        networks: ["192.168.10.1/24"]
router 50641e0d-bcb0-4925-a5ff-8360d178ba47 (neutron-0fd652d1-f17d-464d-9b1c-68fd88dec5c3) (aka router1)
    port lrp-367ac68a-9732-4c91-a458-2fc221491e2f
        mac: "fa:16:3e:f0:55:63"
        networks: ["10.0.0.1/26"]
    port lrp-55edcb76-e904-4dea-8ddb-cb770c3b7aee
        mac: "fa:16:3e:ed:a2:74"
        networks: ["fded:d86e:b73f::1/64"]
    port lrp-59f74080-ff5f-474d-9910-f362122d0ddf
        mac: "fa:16:3e:40:6a:5f"
        networks: ["172.24.4.19/24", "2001:db8::1/64"]
        gateway chassis: [949fe1d5-3db2-49dd-a8b7-7bc7fa119d97]
    po...

[Daniel Alvarez (dalvarezs)]

Looks like the reason that it worked manually for me is because I had some other ports in the common network for the two routers. I set a breakpoint in the test here [0] and ran ovn-trace simulating a packet from VM1 to VM2 setting the dst mac address as the gateway for VM1:

# ip,reg14=0x2,vlan_tci=0x0000,dl_src=fa:16:3e:5d:31:2c,dl_dst=fa:16:3e:50:ec:9e,nw_src=10.10.210.10,nw_dst=10.10.220.3,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=32

ingress(dp="tempest-test-network--158358908", inport="148e57")
--------------------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:4066): inport == "148e57" && eth.src == {fa:16:3e:5d:31:2c}, priority 50, uuid 7010effd
    next;
 1. ls_in_port_sec_ip (ovn-northd.c:2821): inport == "148e57" && eth.src == fa:16:3e:5d:31:2c && ip4.src == {10.10.210.10}, priority 90, uuid 4d34efba
    next;
 3. ls_in_pre_acl (ovn-northd.c:3198): ip, priority 100, uuid f10ab6da
    reg0[0] = 1;
    next;
 5. ls_in_pre_stateful (ovn-northd.c:3325): reg0[0] == 1, priority 100, uuid 7e1f694b
    ct_next;

ct_next(ct_state=est|trk /* default (use --ct to customize) */)
---------------------------------------------------------------
 6. ls_in_acl (ovn-northd.c:3512): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (inport == @pg_506c79a9_bf70_4ada_9f07_d6803b660821 && ip4), priority 2002, uuid 280cd06e
    next;
16. ls_in_l2_lkup (ovn-northd.c:4441): eth.dst == fa:16:3e:50:ec:9e, priority 50, uuid 228d180f
    outport = "affa4a";
    output;

egress(dp="tempest-test-network--158358908", inport="148e57", outport="affa4a")
-------------------------------------------------------------------------------
 1. ls_out_pre_acl (ovn-northd.c:3154): ip && outport == "affa4a", priority 110, uuid a946e866
    next;
 9. ls_out_port_sec_l2 (ovn-northd.c:4524): outport == "affa4a", priority 50, uuid bb820d91
    output;
    /* output to "affa4a", type "patch" */

ingress(dp="tempest-ap1_rt-1260629871", inport="lrp-affa4a")
------------------------------------------------------------
 0. lr_in_admission (ovn-northd.c:5070): eth.dst == fa:16:3e:50:ec:9e && inport == "lrp-affa4a", priority 50, uuid 678b5eab
    next;
 7. lr_in_ip_routing (ovn-northd.c:4652): ip4.dst == 10.10.220.0/24, priority 49, uuid 0320c941
    ip.ttl--;
    reg0 = 10.10.200.5;
    reg1 = 10.10.200.8;
    eth.src = fa:16:3e:26:a9:4b;
    outport = "lrp-dfd78c";
    flags.loopback = 1;
    next;
 8. lr_in_arp_resolve (ovn-northd.c:6470): outport == "lrp-dfd78c" && reg0 == 10.10.200.5, priority 100, uuid 1e85b002
    eth.dst = fa:16:3e:51:8d:06;
    next;
10. lr_in_arp_request (ovn-northd.c:6566): 1, priority 0, uuid 41a990d6
    output;

egress(dp="tempest-ap1_rt-1260629871", inport="lrp-affa4a", outport="lrp-dfd78c")
---------------------------------------------------------------------------------
 3. lr_out_delivery (ovn-northd.c:6594): outport == "lrp-dfd78c", priority 100, uuid bd19462c
    output;
    /* output to "lrp-dfd78c", type "patch" */

ingress(dp="tempest-test-network--641229389", inport="dfd78c")
--------------------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:4066): inport == "dfd78c", priority 50, ...

[Daniel Alvarez (dalvarezs)]

After more investigation, the problem is that the intermediate network ports get created as normal ports and then added to the router. When added to the router they get their owner changed to network:router so become trusted ports. networking-ovn doesn't honour this change of owner and they remain in the default drop port group so all the traffic is dropped.

If another port existed in such network, the Logical Flows corresponding to stateful (allow-related) ACLs get installed and traffic works.

The proper fix is patch the update_port method to honour changes to port owner.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Daniel Alvarez (<email address hidden>) on branch: master
Review: https://review.openstack.org/617305

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617307
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=605ad27f053a1ef9ffb3601c6023987ddb6a22e1
Submitter: Zuul
Branch: master

commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617979

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617979
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=e83edbc171a33fa5f495b5b9fe483988419f2bfd
Submitter: Zuul
Branch: stable/rocky

commit e83edbc171a33fa5f495b5b9fe483988419f2bfd
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/681568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681568
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=49e6e141f254762f63ba8347ead84e28eefca838
Submitter: Zuul
Branch: stable/queens

commit 49e6e141f254762f63ba8347ead84e28eefca838
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 4.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 5.1.0 release.