[floatingip port_forwarding] changing external port to used value hangs with retriable exception

Bug #1878299 reported by Flavio Fernandes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Flavio Fernandes

Bug Description

When trying to change external-protocol-port to a value that is already used by another pf row, there is no error checking and we end up 'stuck' until the api times out. The neutron db is catching
the improper config, but the validation should likely not allow it to get that far.

(overcloud) [stack@undercloud-0 ~]$ openstack floating ip port forwarding list ${FIP_UUID}
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| ID | Internal Port ID | Internal IP Address | Internal Port | External Port | Protocol |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
| 5a8515b8-9e03-442f-a8d7-41acd11f59b5 | 63767606-35ea-4c08-b6c8-1216d0c407e8 | 192.168.30.159 | 22 | 2021 | tcp |
| 60693ea5-e985-404d-87ce-798dd4f6f4da | d5a31eba-89cb-40a8-ba98-d6ca1a8fffb1 | 192.168.30.84 | 22 | 2020 | tcp |
+--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$

# now, doing something invalid: try to change external-protocol-port of $PF_ID2 to the port that is being used by $PF_ID (60693ea5-e985-404d-87ce-798dd4f6f4da)

(overcloud) [stack@undercloud-0 ~]$ echo ${PF_ID2}
5a8515b8-9e03-442f-a8d7-41acd11f59b5

(overcloud) [stack@undercloud-0 ~]$ time openstack floating ip port forwarding set --external-protocol-port 2020 ${FIP_UUID} ${PF_ID2}

HttpException: 504: Server Error for url: http://10.0.0.125:9696/v2.0/floatingips/aee2c979-31a1-453a-b508-319c71fee9dc/port_forwardings/5a8515b8-9e03-442f-a8d7-41acd11f59b5, The server didn't respond in time.: 504 Gateway Time-out

real 2m3.552s
user 0m1.672s
sys 0m0.727s

=-==-

[root@controller-0 neutron]# tail -F server.log
...
2020-05-12 17:54:15.900 26 DEBUG neutron.api.v2.base [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body: {'port_forwarding': {'external_port': 2020}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:16.137 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57154) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:16.370 26 DEBUG neutron_lib.db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key 'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id': '5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped /usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:16.371 26 DEBUG oslo_db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for function neutron.api.v2.base.Controller._update wrapper /usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:16.458 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.141', 48380) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.882 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84', 39608) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:17.946 26 DEBUG neutron.api.v2.base [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body: {'port_forwarding': {'external_port': 2020}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2020-05-12 17:54:18.145 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38', 57244) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:18.292 26 DEBUG neutron_lib.db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key 'uniq_port_forwardings0floatingip_id0external_port0protocol'")
[SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE portforwardings.id = %(portforwardings_id)s]
[parameters: {'external_port': 2020, 'portforwardings_id': '5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
(Background on this error at: http://sqlalche.me/e/gkpj) wrapped /usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
2020-05-12 17:54:18.292 26 DEBUG oslo_db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for function neutron.api.v2.base.Controller._update wrapper /usr/lib/python3.6/site-packages/oslo_db/api.py:156
2020-05-12 17:54:20.474 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.141', 48566) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.027 27 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-8005a4d6-5356-4472-aa28-6debbf9f1175 b4758ff03a5e44b8b23e758ca548d08d 8b966122dec44bed80b5c4fd53919a0d - default default] Attributes excluded by policy engine: ['shared'] _exclude_attributes_by_policy /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256
2020-05-12 17:54:22.166 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57438) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:22.481 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48664) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:23.903 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84', 39922) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.175 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57534) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:24.489 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48760) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:25.910 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.84', 40016) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.183 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38', 57626) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:26.494 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48854) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2020-05-12 17:54:28.195 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57722) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
...

Revision history for this message
Flavio Fernandes (ffernand) wrote :
Revision history for this message
Flavio Fernandes (ffernand) wrote :
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
tags: added: api l3-dvr-backlog
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/738145

Changed in neutron:
assignee: nobody → Flavio Fernandes (ffernand)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/738145
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f3797403486ba43100ff9ec8917d38c8723e12b2
Submitter: Zuul
Branch: master

commit f3797403486ba43100ff9ec8917d38c8723e12b2
Author: Flavio Fernandes <email address hidden>
Date: Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update

    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).

    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744177

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/744182

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/744186

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/744188

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/744993

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/744993
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5f9b5dd86161608599696a78e033ab0c212c4615
Submitter: Zuul
Branch: master

commit 5f9b5dd86161608599696a78e033ab0c212c4615
Author: Flavio Fernandes <email address hidden>
Date: Wed Aug 5 17:10:27 2020 -0400

    port_forwarding: validate args before invoking db update (cont.)

    This is a follow up for https://review.opendev.org/#/c/738145/

    During backporting review, it became clear that unit test had a
    flaw. It assumed that order of items in dictionary that make up
    the exception message did not change. That is not true, based
    on the python version used.

    This follow up also includes a review feedback that did not make
    into the original change: rename function that raises exception
    to have "raise" in its name (raise_port_forwarding_update_failed).

    Change-Id: I6fcd64e205e584017e6c9022f82a5497ea1cc576
    Closes-Bug: #1878299

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/744182
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9273af2c50b03bb5b1d7fb1944d0271e2785b35b
Submitter: Zuul
Branch: stable/train

commit 9273af2c50b03bb5b1d7fb1944d0271e2785b35b
Author: Flavio Fernandes <email address hidden>
Date: Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update

    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).

    This is a backport that combines changes from 2 changes that go together:
    https://review.opendev.org/#/c/738145/
    https://review.opendev.org/#/c/744993/

    Note: pep8 failed with following error in stable/train:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:237:9:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:261:9:
        N322 Possible use of no-op mock method. please use assert_called_once_with.
            mock_pf_get_objects.assert_called_once()
            ^
    So additional changes were needed for backport.

    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299
    (cherry picked from commit f3797403486ba43100ff9ec8917d38c8723e12b2)
    (cherry picked from commit 838399f0a489e898cfefe49dbe026553bc117381)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/744188
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4ac33d1a9bf1655ae1316f8c7b2ad3047abba69c
Submitter: Zuul
Branch: stable/rocky

commit 4ac33d1a9bf1655ae1316f8c7b2ad3047abba69c
Author: Flavio Fernandes <email address hidden>
Date: Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update

    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).

    This is a backport that combines changes from 2 changes that go together:
    https://review.opendev.org/#/c/738145/
    https://review.opendev.org/#/c/744993/

    Note: pep8 failed with following error:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:237:9:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:261:9:
        N322 Possible use of no-op mock method. please use assert_called_once_with.
            mock_pf_get_objects.assert_called_once()
            ^
    So additional changes were needed for backport.

    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299
    (cherry picked from commit f3797403486ba43100ff9ec8917d38c8723e12b2)
    (cherry picked from commit 838399f0a489e898cfefe49dbe026553bc117381)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/744177
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c678d89dabc0b39a266c7c0c7b99bd156b6fa5c9
Submitter: Zuul
Branch: stable/ussuri

commit c678d89dabc0b39a266c7c0c7b99bd156b6fa5c9
Author: Flavio Fernandes <email address hidden>
Date: Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update

    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).

    This is a backport that combines changes from 2 changes that go together:
    https://review.opendev.org/#/c/738145/
    https://review.opendev.org/#/c/744993/

    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299
    (cherry picked from commit f3797403486ba43100ff9ec8917d38c8723e12b2)
    (cherry picked from commit 838399f0a489e898cfefe49dbe026553bc117381)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/744186
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b3647782e4dae56529bbf59918a51326df47b32e
Submitter: Zuul
Branch: stable/stein

commit b3647782e4dae56529bbf59918a51326df47b32e
Author: Flavio Fernandes <email address hidden>
Date: Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update

    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).

    This is a backport that combines changes from 2 changes that go together:
    https://review.opendev.org/#/c/738145/
    https://review.opendev.org/#/c/744993/

    Note: pep8 failed with following error:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:237:9:
      ./neutron/tests/unit/services/portforwarding/test_pf_plugin.py:261:9:
        N322 Possible use of no-op mock method. please use assert_called_once_with.
            mock_pf_get_objects.assert_called_once()
            ^
    So additional changes were needed for backport.

    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299
    (cherry picked from commit f3797403486ba43100ff9ec8917d38c8723e12b2)
    (cherry picked from commit 838399f0a489e898cfefe49dbe026553bc117381)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol

This issue was fixed in the openstack/neutron rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.