Unstable unit test uses subnet broadcast address

Bug #1821948 reported by Bence Romsics
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Bence Romsics

Bug Description

This is a low frequency gate failure in unit tests.

Example log:
http://logs.openstack.org/10/645210/4/check/openstack-tox-py37/688ffa8/job-output.txt.gz

Logstash search:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22line%20171,%20in%20test_port_ip_update_revises%5C%22%20AND%20voting:1&from=864000s

2019-03-25 12:16:24.333688 | ubuntu-bionic | ==============================
2019-03-25 12:16:24.333764 | ubuntu-bionic | Failed 1 tests - output below:
2019-03-25 12:16:24.333837 | ubuntu-bionic | ==============================
2019-03-25 12:16:24.333863 | ubuntu-bionic |
2019-03-25 12:16:24.334052 | ubuntu-bionic | neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises
2019-03-25 12:16:24.334243 | ubuntu-bionic | ---------------------------------------------------------------------------------------------------------
2019-03-25 12:16:24.334271 | ubuntu-bionic |
2019-03-25 12:16:24.334326 | ubuntu-bionic | Captured traceback:
2019-03-25 12:16:24.334381 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~
2019-03-25 12:16:24.334471 | ubuntu-bionic | b'Traceback (most recent call last):'
2019-03-25 12:16:24.334662 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 174, in func'
2019-03-25 12:16:24.334754 | ubuntu-bionic | b' return f(self, *args, **kwargs)'
2019-03-25 12:16:24.335103 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/services/revisions/test_revision_plugin.py", line 171, in test_port_ip_update_revises'
2019-03-25 12:16:24.335243 | ubuntu-bionic | b" response = self._update('ports', port['port']['id'], new)"
2019-03-25 12:16:24.335490 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 601, in _update'
2019-03-25 12:16:24.335642 | ubuntu-bionic | b' self.assertEqual(expected_code, res.status_int)'
2019-03-25 12:16:24.335921 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual'
2019-03-25 12:16:24.336035 | ubuntu-bionic | b' self.assertThat(observed, matcher, message)'
2019-03-25 12:16:24.336297 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat'
2019-03-25 12:16:24.336372 | ubuntu-bionic | b' raise mismatch_error'
2019-03-25 12:16:24.336486 | ubuntu-bionic | b'testtools.matchers._impl.MismatchError: 200 != 400'
2019-03-25 12:16:24.336523 | ubuntu-bionic | b''
2019-03-25 12:16:24.336549 | ubuntu-bionic |
2019-03-25 12:16:24.336599 | ubuntu-bionic | Captured stderr:
2019-03-25 12:16:24.336650 | ubuntu-bionic | ~~~~~~~~~~~~~~~~
2019-03-25 12:16:24.337086 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.'
2019-03-25 12:16:24.337157 | ubuntu-bionic | b' DeprecationWarning)'
2019-03-25 12:16:24.337594 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.'
2019-03-25 12:16:24.337664 | ubuntu-bionic | b' DeprecationWarning)'
2019-03-25 12:16:24.337701 | ubuntu-bionic | b''

With some extra debug logging added I managed to obtain this error message:

ERROR [neutron.tests.unit.db.test_db_base_plugin_v2] XXX b\'{"NeutronError": {"type": "InvalidIpForNetwork", "message": "IP address 10.0.0.255 is not a valid IP for any of the subnets on the specified network.", "detail": ""}}\'

Reading the unit test source it seems likely that a random IP+1 is occasionally the subnet broadcast address which is invalid as a fixed_ip.

https://opendev.org/openstack/neutron/src/commit/1ea9326fda303b48905d7f7748d320ba8e9322aa/neutron/tests/unit/services/revisions/test_revision_plugin.py#L169

I'm going to upload an attempted fix soon.

Tags: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/648172

Changed in neutron:
status: New → In Progress
Revision history for this message
Bence Romsics (bence-romsics) wrote :

Reproduce with:

success=0 ; fail=0 ; for i in $( seq 1 100 ) ; do if tox -e py35 neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises ; then (( success++ )) ; else (( fail++ )) ; fi ; done ; echo $success $fail

Choose a big enough count, 100 is usually big enough to have 1 or 2 failures.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/648172
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=765691d1e28fe59f6981cd9e24d57e358163f499
Submitter: Zuul
Branch: master

commit 765691d1e28fe59f6981cd9e24d57e358163f499
Author: Bence Romsics <email address hidden>
Date: Wed Mar 27 16:37:51 2019 +0100

    Do not use subnet broadcast address in unit test

    neutron.tests.unit.services.revisions.test_revision_plugin.\
        TestRevisionPlugin.test_port_ip_update_revises
    occasionally fails in the gate, because it assumes that a random
    IP+1 is also a valid IP. However the +1 address may be the subnet
    broadcast address which is then rejected by neutron.

    This change makes sure to use a free IP from the subnet instead of
    the +1 address.

    Change-Id: I1b27cac301cdc1acf426b0e71c037c157e93cd45
    Closes-Bug: #1821948

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.