func: neutron.agent.linux.utils.ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: Unable to create lock file /run/ebtables.lock.

Bug #1697833 reported by Akihiro Motoki
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Kevin Benton

Bug Description

http://logs.openstack.org/21/473721/2/gate/gate-neutron-dsvm-functional-ubuntu-xenial/788a8a8/testr_results.html.gz

neutron.tests.functional.agent.linux.test_linuxbridge_arp_protect.LinuxBridgeARPSpoofTestCase
test_arp_correct_protection

traceback-1: {{{
Traceback (most recent call last):
  File "neutron/tests/functional/agent/linux/test_linuxbridge_arp_protect.py", line 44, in _ensure_rules_cleaned
    self.assertEqual([], rules, 'Test leaked ebtables rules')
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = []
actual = [u'-i test-veth0d943e -j neutronMAC-test-veth0d943e',
 u'-i test-veth0d943e --among-src fa:16:3e:7f:c8:97, -j RETURN ',
 u'-p ARP --arp-ip-src 192.168.0.1 -j ACCEPT ']
: Test leaked ebtables rules
}}}

Traceback (most recent call last):
  File "neutron/tests/base.py", line 118, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/agent/linux/test_linuxbridge_arp_protect.py", line 62, in test_arp_correct_protection
    self._add_arp_protection(self.source, [self.source.ip])
  File "neutron/tests/functional/agent/linux/test_linuxbridge_arp_protect.py", line 53, in _add_arp_protection
    arp_protect.setup_arp_spoofing_protection(name, port_dict)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py", line 57, in setup_arp_spoofing_protection
    install_arp_spoofing_protection(vif, addresses, current_rules)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
    return f(*args, **kwargs)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py", line 114, in install_arp_spoofing_protection
    vif_chain, '-p', 'ARP'])
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py", line 194, in ebtables
    return execute(['ebtables', '--concurrent'] + comm, run_as_root=True)
  File "neutron/agent/linux/ip_lib.py", line 900, in execute
    log_fail_as_error=log_fail_as_error, **kwargs)
  File "neutron/agent/linux/utils.py", line 151, in execute
    raise ProcessExecutionError(msg, returncode=returncode)
neutron.agent.linux.utils.ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: Unable to create lock file /run/ebtables.lock.

Akihiro Motoki (amotoki)
Changed in neutron:
importance: Undecided → Critical
Changed in neutron:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/474063

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/474063
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2e7b787f0e60d3707fe4dec7f8acbc90cf62ea29
Submitter: Jenkins
Branch: master

commit 2e7b787f0e60d3707fe4dec7f8acbc90cf62ea29
Author: Kevin Benton <email address hidden>
Date: Tue Jun 13 22:33:24 2017 -0700

    Retry ebtables lock acquisition failures

    It seems after the merge of
    https://bugs.launchpad.net/ubuntu/+source/ebtables/+bug/1645324
    that ebtables can fail to acquire a lock and bail with an error
    255. This adds some retry logic to retry it up to 10 times to
    work around this issue.

    Closes-Bug: #1697833
    Change-Id: Ic9dcf4b236a93e8811413c6ce2c4b82602544c6d

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/474303

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/474305

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/474303
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b67e63e97073196658166908336cdf1ab0a581a
Submitter: Jenkins
Branch: stable/ocata

commit 7b67e63e97073196658166908336cdf1ab0a581a
Author: Kevin Benton <email address hidden>
Date: Tue Jun 13 22:33:24 2017 -0700

    Retry ebtables lock acquisition failures

    It seems after the merge of
    https://bugs.launchpad.net/ubuntu/+source/ebtables/+bug/1645324
    that ebtables can fail to acquire a lock and bail with an error
    255. This adds some retry logic to retry it up to 10 times to
    work around this issue.

    Closes-Bug: #1697833
    Change-Id: Ic9dcf4b236a93e8811413c6ce2c4b82602544c6d
    (cherry picked from commit 2e7b787f0e60d3707fe4dec7f8acbc90cf62ea29)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/474305
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b5b68b3752c0bc1d58d35eddd9bf3dae95550261
Submitter: Jenkins
Branch: stable/newton

commit b5b68b3752c0bc1d58d35eddd9bf3dae95550261
Author: Kevin Benton <email address hidden>
Date: Tue Jun 13 22:33:24 2017 -0700

    Retry ebtables lock acquisition failures

    It seems after the merge of
    https://bugs.launchpad.net/ubuntu/+source/ebtables/+bug/1645324
    that ebtables can fail to acquire a lock and bail with an error
    255. This adds some retry logic to retry it up to 10 times to
    work around this issue.

    Conflicts:
        neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py

    Closes-Bug: #1697833
    Change-Id: Ic9dcf4b236a93e8811413c6ce2c4b82602544c6d
    (cherry picked from commit 2e7b787f0e60d3707fe4dec7f8acbc90cf62ea29)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/473993
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6451e0750356b600ac01c13ac86eb556b2a616ee
Submitter: Jenkins
Branch: master

commit 6451e0750356b600ac01c13ac86eb556b2a616ee
Author: Brian Haley <email address hidden>
Date: Tue Jun 13 18:14:43 2017 -0400

    Fix linuxbridge ebtables locking

    Change linuxbridge ARP protection code to always hold the
    neutron ebtables lock while operating on rules. There
    were a couple of places where we were reading the rules
    without it.

    Change-Id: Id7b38a3a662fb2e2736baa015492c36699319e58
    Related-bug: #1697833

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.0.0b3

This issue was fixed in the openstack/neutron 11.0.0.0b3 development milestone.

tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.4.1

This issue was fixed in the openstack/neutron 9.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.3

This issue was fixed in the openstack/neutron 10.0.3 release.

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.