Unstable unit test uses subnet broadcast address

[Bence Romsics]

This is a low frequency gate failure in unit tests.

Example log:
http://logs.openstack.org/10/645210/4/check/openstack-tox-py37/688ffa8/job-output.txt.gz

Logstash search:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22line%20171,%20in%20test_port_ip_update_revises%5C%22%20AND%20voting:1&from=864000s

2019-03-25 12:16:24.333688 | ubuntu-bionic | ==============================
2019-03-25 12:16:24.333764 | ubuntu-bionic | Failed 1 tests - output below:
2019-03-25 12:16:24.333837 | ubuntu-bionic | ==============================
2019-03-25 12:16:24.333863 | ubuntu-bionic |
2019-03-25 12:16:24.334052 | ubuntu-bionic | neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises
2019-03-25 12:16:24.334243 | ubuntu-bionic | ---------------------------------------------------------------------------------------------------------
2019-03-25 12:16:24.334271 | ubuntu-bionic |
2019-03-25 12:16:24.334326 | ubuntu-bionic | Captured traceback:
2019-03-25 12:16:24.334381 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~
2019-03-25 12:16:24.334471 | ubuntu-bionic | b'Traceback (most recent call last):'
2019-03-25 12:16:24.334662 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 174, in func'
2019-03-25 12:16:24.334754 | ubuntu-bionic | b' return f(self, *args, **kwargs)'
2019-03-25 12:16:24.335103 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/services/revisions/test_revision_plugin.py", line 171, in test_port_ip_update_revises'
2019-03-25 12:16:24.335243 | ubuntu-bionic | b" response = self._update('ports', port['port']['id'], new)"
2019-03-25 12:16:24.335490 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 601, in _update'
2019-03-25 12:16:24.335642 | ubuntu-bionic | b' self.assertEqual(expected_code, res.status_int)'
2019-03-25 12:16:24.335921 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual'
2019-03-25 12:16:24.336035 | ubuntu-bionic | b' self.assertThat(observed, matcher, message)'
2019-03-25 12:16:24.336297 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat'
2019-03-25 12:16:24.336372 | ubuntu-bionic | b' raise mismatch_error'
2019-03-25 12:16:24.336486 | ubuntu-bionic | b'testtools.matchers._impl.MismatchError: 200 != 400'
2019-03-25 12:16:24.336523 | ubuntu-bionic | b''
2019-03-25 12:16:24.336549 | ubuntu-bionic |
2019-03-25 12:16:24.336599 | ubuntu-bionic | Captured stderr:
2019-03-25 12:16:24.336650 | ubuntu-bionic | ~~~~~~~~~~~~~~~~
2019-03-25 12:16:24.337086 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.'
2019-03-25 12:16:24.337157 | ubuntu-bionic | b' DeprecationWarning)'
2019-03-25 12:16:24.337594 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.'
2019-03-25 12:16:24.337664 | ubuntu-bionic | b' DeprecationWarning)'
2019-03-25 12:16:24.337701 | ubuntu-bionic | b''

With some extra debug logging added I managed to obtain this error message:

ERROR [neutron.tests.unit.db.test_db_base_plugin_v2] XXX b\'{"NeutronError": {"type": "InvalidIpForNetwork", "message": "IP address 10.0.0.255 is not a valid IP for any of the subnets on the specified network.", "detail": ""}}\'

Reading the unit test source it seems likely that a random IP+1 is occasionally the subnet broadcast address which is invalid as a fixed_ip.

https://opendev.org/openstack/neutron/src/commit/1ea9326fda303b48905d7f7748d320ba8e9322aa/neutron/tests/unit/services/revisions/test_revision_plugin.py#L169

I'm going to upload an attempted fix soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/648172

[Bence Romsics (bence-romsics)]

Reproduce with:

success=0 ; fail=0 ; for i in $( seq 1 100 ) ; do if tox -e py35 neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises ; then (( success++ )) ; else (( fail++ )) ; fi ; done ; echo $success $fail

Choose a big enough count, 100 is usually big enough to have 1 or 2 failures.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648172
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=765691d1e28fe59f6981cd9e24d57e358163f499
Submitter: Zuul
Branch: master

commit 765691d1e28fe59f6981cd9e24d57e358163f499
Author: Bence Romsics <email address hidden>
Date: Wed Mar 27 16:37:51 2019 +0100

    Do not use subnet broadcast address in unit test

    neutron.tests.unit.services.revisions.test_revision_plugin.\
        TestRevisionPlugin.test_port_ip_update_revises
    occasionally fails in the gate, because it assumes that a random
    IP+1 is also a valid IP. However the +1 address may be the subnet
    broadcast address which is then rejected by neutron.

    This change makes sure to use a free IP from the subnet instead of
    the +1 address.

    Change-Id: I1b27cac301cdc1acf426b0e71c037c157e93cd45
    Closes-Bug: #1821948

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.