Floating up update is deleting the NAT entry in OVN North db

Bug #1833820 reported by Numan Siddique on 2019-06-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
High
Li Zhouzhou

Bug Description

Below are the commands and the output.

stack@test-octavia-2:~/networking-ovn$ openstack floating ip create --port p1 public
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2019-06-22T16:07:31Z |
| description | |
| dns_domain | |
| dns_name | |
| fixed_ip_address | 10.0.0.7 |
| floating_ip_address | 172.24.4.84 |
| floating_network_id | 8987dc42-a9f3-4962-99cb-46a7a94908fc |
| id | 7876f69b-ee59-43d4-b5a0-036753eb2fcc |
| location | Munch({'project': Munch({'domain_id': 'default', 'id': u'5c01831e2d77442caa373b84af10c9c7', 'name': 'admin', 'domain_name': None}), 'cloud': '', 'region_name': 'RegionOne', 'zone': None}) |
| name | 172.24.4.84 |
| port_details | None |
| port_id | 8534dead-2b52-46c0-bf09-eded43aee518 |
| project_id | 5c01831e2d77442caa373b84af10c9c7 |
| qos_policy_id | None |
| revision_number | 0 |
| router_id | 09e58725-9d2b-4eb3-bdb3-63e630a0ca5d |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| updated_at | 2019-06-22T16:07:31Z |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@test-octavia-2:~/networking-ovn$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 7876f69b-ee59-43d4-b5a0-036753eb2fcc | 172.24.4.84 | 10.0.0.7 | 8534dead-2b52-46c0-bf09-eded43aee518 | 8987dc42-a9f3-4962-99cb-46a7a94908fc | 5c01831e2d77442caa373b84af10c9c7 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
stack@test-octavia-2:~/networking-ovn$ ovn-nbctl list nat
_uuid : 0479938d-2104-4817-a81b-1d7348798381
external_ids : {}
external_ip : "172.24.4.122"
external_mac : []
logical_ip : "10.0.0.0/26"
logical_port : []
type : snat

_uuid : 2df36cba-e6bc-4967-8112-fa78af3c4205
external_ids : {"neutron:fip_id"="7876f69b-ee59-43d4-b5a0-036753eb2fcc", "neutron:fip_port_id"="8534dead-2b52-46c0-bf09-eded43aee518", "neutron:revision_number"="1", "neutron:router_name"="neutron-09e58725-9d2b-4eb3-bdb3-63e630a0ca5d"}
external_ip : "172.24.4.84"
external_mac : []
logical_ip : "10.0.0.7"
logical_port : []
type : dnat_and_snat
stack@test-octavia-2:~/networking-ovn$
stack@test-octavia-2:~/networking-ovn$ openstack floating ip set --port p2 7876f69b-ee59-43d4-b5a0-036753eb2fcc
stack@test-octavia-2:~/networking-ovn$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 7876f69b-ee59-43d4-b5a0-036753eb2fcc | 172.24.4.84 | 10.0.0.16 | c8d7b3e4-f9cd-49c7-ba81-974595840fa7 | 8987dc42-a9f3-4962-99cb-46a7a94908fc | 5c01831e2d77442caa373b84af10c9c7 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
stack@test-octavia-2:~/networking-ovn$ ovn-nbctl list nat
_uuid : 0479938d-2104-4817-a81b-1d7348798381
external_ids : {}
external_ip : "172.24.4.122"
external_mac : []
logical_ip : "10.0.0.0/26"
logical_port : []
type : snat

Changed in networking-ovn:
importance: Undecided → Critical
importance: Critical → High
Li Zhouzhou (lizhouzhou) wrote :

I reproduced it in my environment, but the fip info had been appeared in ovn about 3 min.

Li Zhouzhou (lizhouzhou) on 2019-07-11
Changed in networking-ovn:
assignee: nobody → Li Zhouzhou (lizhouzhou)
Li Zhouzhou (lizhouzhou) wrote :

After read the code, I found it caused by the transaction of 'update_floatingip' in OvnClient. The transaction includes '_delete_floating' and '_create_or_update_floatingip', the '_delete_floating' will execute after '_update_floating', so the dnat_and_snat rule disapperd in OVN.
I and Yunxiang modify the logic of transaction, we split it into two parts, it worked as we expected. The OVN will produce the dnat_and_snat rule immediately.

As talked about this function, I don't know why '_delete_floating' is needed in 'update_floatingip', I think we can remove it.

Fix proposed to branch: master
Review: https://review.opendev.org/670219

Changed in networking-ovn:
status: New → In Progress

Change abandoned by Li Zhouzhou (<email address hidden>) on branch: master
Review: https://review.opendev.org/670219
Reason: Daniel have been solved it.
https://review.opendev.org/#/c/670868/

Reviewed: https://review.opendev.org/670868
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=d662f444280f5dc0a304a0b4de4767c91304c747
Submitter: Zuul
Branch: master

commit d662f444280f5dc0a304a0b4de4767c91304c747
Author: Daniel Alvarez <email address hidden>
Date: Mon Jul 15 17:29:03 2019 +0200

    Always add NAT rule to a LR when updating a FIP

    Before this patch, the NAT rule was attempted to be updated when a
    FIP was reassigned to another port. However, this was a noop
    since the NAT rule didn't exist anymore causing the reassigning to
    be ineffective and to fail silently.

    This patch is always adding the NAT rule to the Logical Router
    no matter if the Neutron FIP is being added or updated.

    The bug that this patch addresses was being hit in the gate with
    around a 30% ratio because the maintenance task was fixing it while
    tempest was still rying to SSH into it.

    Change-Id: Icebf4a82f64989112c3ca810b4358de490108c2d
    Closes-Bug: #1835029
    Closes-Bug: #1833820
    Co-Authored-By: Jakub Libosvar <email address hidden>
    Signed-off-by: Daniel Alvarez <email address hidden>

Changed in networking-ovn:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/671969
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=13689522d6b97b4d2246e5de6af0447b39ec50ed
Submitter: Zuul
Branch: stable/stein

commit 13689522d6b97b4d2246e5de6af0447b39ec50ed
Author: Daniel Alvarez <email address hidden>
Date: Mon Jul 15 17:29:03 2019 +0200

    Always add NAT rule to a LR when updating a FIP

    Before this patch, the NAT rule was attempted to be updated when a
    FIP was reassigned to another port. However, this was a noop
    since the NAT rule didn't exist anymore causing the reassigning to
    be ineffective and to fail silently.

    This patch is always adding the NAT rule to the Logical Router
    no matter if the Neutron FIP is being added or updated.

    The bug that this patch addresses was being hit in the gate with
    around a 30% ratio because the maintenance task was fixing it while
    tempest was still rying to SSH into it.

    Conflicts:
     networking_ovn/common/ovn_client.py

    Change-Id: Icebf4a82f64989112c3ca810b4358de490108c2d
    Closes-Bug: #1835029
    Closes-Bug: #1833820
    Co-Authored-By: Jakub Libosvar <email address hidden>
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit d662f444280f5dc0a304a0b4de4767c91304c747)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/671970
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=9490a84f21eef0c112c625227748a61738334d7a
Submitter: Zuul
Branch: stable/rocky

commit 9490a84f21eef0c112c625227748a61738334d7a
Author: Daniel Alvarez <email address hidden>
Date: Mon Jul 15 17:29:03 2019 +0200

    Always add NAT rule to a LR when updating a FIP

    Before this patch, the NAT rule was attempted to be updated when a
    FIP was reassigned to another port. However, this was a noop
    since the NAT rule didn't exist anymore causing the reassigning to
    be ineffective and to fail silently.

    This patch is always adding the NAT rule to the Logical Router
    no matter if the Neutron FIP is being added or updated.

    The bug that this patch addresses was being hit in the gate with
    around a 30% ratio because the maintenance task was fixing it while
    tempest was still rying to SSH into it.

    Conflicts:
     networking_ovn/common/ovn_client.py

    Change-Id: Icebf4a82f64989112c3ca810b4358de490108c2d
    Closes-Bug: #1835029
    Closes-Bug: #1833820
    Co-Authored-By: Jakub Libosvar <email address hidden>
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit d662f444280f5dc0a304a0b4de4767c91304c747)
    (cherry picked from commit 13689522d6b97b4d2246e5de6af0447b39ec50ed)

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/671971
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=fcc79fe9f7539b5665847f9edd8aa8cb37b6dc0f
Submitter: Zuul
Branch: stable/queens

commit fcc79fe9f7539b5665847f9edd8aa8cb37b6dc0f
Author: Daniel Alvarez <email address hidden>
Date: Mon Jul 15 17:29:03 2019 +0200

    Always add NAT rule to a LR when updating a FIP

    Before this patch, the NAT rule was attempted to be updated when a
    FIP was reassigned to another port. However, this was a noop
    since the NAT rule didn't exist anymore causing the reassigning to
    be ineffective and to fail silently.

    This patch is always adding the NAT rule to the Logical Router
    no matter if the Neutron FIP is being added or updated.

    The bug that this patch addresses was being hit in the gate with
    around a 30% ratio because the maintenance task was fixing it while
    tempest was still rying to SSH into it.

    Conflicts:
     networking_ovn/common/ovn_client.py

    Change-Id: Icebf4a82f64989112c3ca810b4358de490108c2d
    Closes-Bug: #1835029
    Closes-Bug: #1833820
    Co-Authored-By: Jakub Libosvar <email address hidden>
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit d662f444280f5dc0a304a0b4de4767c91304c747)
    (cherry picked from commit 13689522d6b97b4d2246e5de6af0447b39ec50ed)
    (cherry picked from commit 9490a84f21eef0c112c625227748a61738334d7a)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers