neutron

Unable to create DVR+HA routers

Bug #1560945 reported by John Schwarz on 2016-03-23

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	John Schwarz	neutron newton-1 "n1"

Bug Description

When creating a new DVR+HA, the router is created (the API returns successfully) but the l3 agent enters an endless loop:

2016-03-23 13:57:37.340 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'a04b3fd7-d46c-4520-82af-18d16835469d'
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/agent.py", line 497, in _process_router_update
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/agent.py", line 436, in _process_router_if_compatible
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent self._process_updated_router(router)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/agent.py", line 450, in _process_updated_router
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent ri.process(self)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/dvr_edge_ha_router.py", line 92, in process
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent super(DvrEdgeHaRouter, self).process(agent)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/dvr_local_router.py", line 486, in process
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/dvr_router_base.py", line 30, in process
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/ha_router.py", line 386, in process
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent super(HaRouter, self).process(agent)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/common/utils.py", line 377, in call
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent self.logger(e)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent self.force_reraise()
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/common/utils.py", line 374, in call
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/router_info.py", line 963, in process
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent self.process_address_scope()
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/neutron/agent/l3/dvr_edge_router.py", line 235, in process_address_scope
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent with snat_iptables_manager.defer_apply():
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'defer_apply'
2016-03-23 13:57:37.340 TRACE neutron.agent.l3.agent

This happens in upstream master.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-23: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/296394

Changed in neutron:
assignee:	nobody → John Schwarz (jschwarz)
status:	New → In Progress

Armando Migliaccio (armando-migliaccio) on 2016-03-23

Changed in neutron:
importance:	Undecided → High
milestone:	none → newton-1
tags:	added: mitaka-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-23: Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/296630

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-23: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/296394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a8b60671150ac383c6ed24c26e773a97a476f7d2
Submitter: Jenkins
Branch: master

commit a8b60671150ac383c6ed24c26e773a97a476f7d2
Author: John Schwarz <email address hidden>
Date: Wed Mar 23 14:05:37 2016 +0200

Fix reference to uninitialized iptables manager

    DvrEdgeRouter.process_address_scope() currently assumes that
    snat_iptables_manager was initialized, however this is only done when an
    external gateway is added. In case a new DVR+HA router was created
    without an external gateway, the l3 agent will raise an exception and
    will not create the router correctly. This patch adds a simple check to
    make sure that it is defined before it's actually used.

Closes-Bug: #1560945
Change-Id: I677e0837956a6d008a3935d961f078987a07d0c4

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-24: Fix merged to neutron (stable/mitaka)

Reviewed: https://review.openstack.org/296630
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=910744781c46e6c200298922cc9c251725ce19fc
Submitter: Jenkins
Branch: stable/mitaka

commit 910744781c46e6c200298922cc9c251725ce19fc
Author: John Schwarz <email address hidden>
Date: Wed Mar 23 14:05:37 2016 +0200

Fix reference to uninitialized iptables manager

Closes-Bug: #1560945

(cherry picked from commit a8b60671150ac383c6ed24c26e773a97a476f7d2)

Change-Id: I677e0837956a6d008a3935d961f078987a07d0c4

tags:

added: in-stable-mitaka

Revision history for this message

Thierry Carrez (ttx) wrote on 2016-03-25: Fix included in openstack/neutron 8.0.0.0rc2

This issue was fixed in the openstack/neutron 8.0.0.0rc2 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-07: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/302570

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-07: Fix merged to neutron (master)

Download full text (11.4 KiB)

Reviewed: https://review.openstack.org/302570
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03ccdd6a6d8e4cea9145e39dd7a1614656024aed
Submitter: Jenkins
Branch: master

commit 3213eb124e40b130e174ac3a91067e2b196788dd
Author: Cedric Brandily <email address hidden>
Date: Tue Mar 29 18:06:44 2016 +0200

Support Routes==2.3

SubMapper.connect allows in Routes<=2.2 to define routes with/without a
name:

map.connect(route_name, route_path)
map.connect(route_path)

    Routes==2.3 SubMapper.connect now requires a route name (but not
    Mapper.connect). This change adds a route name when needed to support
    Routes<=2.2 and Routes==2.3.

    Closes-Bug: #1563028
    Change-Id: I655bfe7f047945896d63dd94c584acfe60d81b38
    (cherry picked from commit c6efa6292cac9267428366dc3cee00f6fc73e208)

commit 4283a7e36b74823f229400f0f049742f630db28b
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Mar 30 11:48:37 2016 +0200

Constraint requirements using mitaka upper-constraints.txt file

Change-Id: I6a908470987ce05769c425c4623df404cef76a10

commit fc690974480611242d4fc6e1deaaf68171fcaed3
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Mar 30 06:21:15 2016 +0000

Imported Translations from Zanata

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I78c24969927224277f381532ad2604cfde8741fc

commit 41be555eddb0f9947fdaa4e73fa74a72677d4d11
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Mar 29 06:07:19 2016 +0000

Imported Translations from Zanata

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Iafa43bfcfca4d6af0d0866206c07fa817f567de6

commit b435ec56af98d2c04de37dd779ed3613ec4583ad
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Mar 28 06:07:02 2016 +0000

Imported Translations from Zanata

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I9e908579b4704c4294367113523b898839586426

commit bec65f6cdd159ec252c4aa0a63aabc9aea8fcfba
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Mar 25 15:25:39 2016 +0900

api tests: Check correct extensions

auto_allocate -> auto-allocated-topology
rbac_policies -> rbac-policies

Closes-Bug: #1561858

(cherry picked from commit 9d3bab6b8d6b2122732faeb66f831927bee0ac7e)
Change-Id: Ied13e52bf78afe571f7243e93005bf5c68fe44db

commit f0c3b19165a5ca73ea8f1dc0ddaad290eee35039
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Mar 25 06:06:36 2016 +0000

Imported Translations from Zanata

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Idb22eac32944a088d24f4d44e537d9bba09dbcc2

commit 99915fa76bd4ce13e0d36059616ba357f4f54ee2
Author: Hynek Mlnarik <email address hidden>
Date: Thu Mar...

Reviewed:  https://review.openstack.org/302570
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03ccdd6a6d8e4cea9145e39dd7a1614656024aed
Submitter: Jenkins
Branch:    master

commit 3213eb124e40b130e174ac3a91067e2b196788dd
Author: Cedric Brandily <zzelle@gmail.com>
Date:   Tue Mar 29 18:06:44 2016 +0200

Support Routes==2.3
    
    SubMapper.connect allows in Routes<=2.2 to define routes with/without a
    name:
    
     map.connect(route_name, route_path)
     map.connect(route_path)
    
    Routes==2.3 SubMapper.connect now requires a route name (but not
    Mapper.connect). This change adds a route name when needed to support
    Routes<=2.2 and Routes==2.3.
    
    Closes-Bug: #1563028
    Change-Id: I655bfe7f047945896d63dd94c584acfe60d81b38
    (cherry picked from commit c6efa6292cac9267428366dc3cee00f6fc73e208)

commit 4283a7e36b74823f229400f0f049742f630db28b
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Wed Mar 30 11:48:37 2016 +0200

Constraint requirements using mitaka upper-constraints.txt file
    
    Change-Id: I6a908470987ce05769c425c4623df404cef76a10

commit fc690974480611242d4fc6e1deaaf68171fcaed3
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Mar 30 06:21:15 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I78c24969927224277f381532ad2604cfde8741fc

commit 41be555eddb0f9947fdaa4e73fa74a72677d4d11
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Mar 29 06:07:19 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Iafa43bfcfca4d6af0d0866206c07fa817f567de6

commit b435ec56af98d2c04de37dd779ed3613ec4583ad
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Mar 28 06:07:02 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I9e908579b4704c4294367113523b898839586426

commit bec65f6cdd159ec252c4aa0a63aabc9aea8fcfba
Author: YAMAMOTO Takashi <yamamoto@midokura.com>
Date:   Fri Mar 25 15:25:39 2016 +0900

api tests: Check correct extensions
    
    auto_allocate -> auto-allocated-topology
    rbac_policies -> rbac-policies
    
    Closes-Bug: #1561858
    
    (cherry picked from commit 9d3bab6b8d6b2122732faeb66f831927bee0ac7e)
    Change-Id: Ied13e52bf78afe571f7243e93005bf5c68fe44db

commit f0c3b19165a5ca73ea8f1dc0ddaad290eee35039
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Mar 25 06:06:36 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Idb22eac32944a088d24f4d44e537d9bba09dbcc2

commit 99915fa76bd4ce13e0d36059616ba357f4f54ee2
Author: Hynek Mlnarik <hmlnarik@redhat.com>
Date:   Thu Mar 24 16:22:17 2016 +0100

Fix setting peer to bridge interfaces
    
    OVSDB implementation refuses to set options:peer column value
     as there is no such column in the Interface table. The correct way
     is to set 'options' column value to a map containing key 'peer', as
     already used in ovs_lib.
    
    Change-Id: Ib5e956f425b36f54cda017c91ac71d9d7ee9747c
    Closes-Bug: 1528894

commit 511875dbce1f39584b00e6293b2bd4cd5d1b1868
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Mar 24 13:47:10 2016 +0000

Updated from global requirements
    
    Change-Id: I1f7955515d74ea00c4730c88d3893259b0beb129

commit 4b86f17db3d122f31c2891249c89bc20d1f6c168
Author: Assaf Muller <amuller@redhat.com>
Date:   Fri Mar 18 16:50:43 2016 -0400

Skip fullstack L3 HA test
    
    It's failing because of a known issue, being handled in:
    https://review.openstack.org/#/c/285480/. That patch will
    unskip the test as part of the fix.
    
    Apart from this test, there's also a known failure in LB
    tests, being handled here:
    and https://review.openstack.org/#/c/294798/
    
    I'm not aware of any other failure modes for fullstack.
    With these two patches merged, we can observe the fullstack
    job failure rate and hopefully make it voting.
    
    Change-Id: I6ea3292605d25848911a2d51c996d10ecd48bf9e
    (cherry picked from commit d6fb0c46bdf238b2b21761e979198388a0446149)

commit 419d44c0d65a3b707f11ac5ecd12924b8469314f
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Mar 24 06:05:22 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I9bbc96164f0e6fdb79fc46ceee9b306b48537bad

commit 910744781c46e6c200298922cc9c251725ce19fc
Author: John Schwarz <jschwarz@redhat.com>
Date:   Wed Mar 23 14:05:37 2016 +0200

Fix reference to uninitialized iptables manager
    
    DvrEdgeRouter.process_address_scope() currently assumes that
    snat_iptables_manager was initialized, however this is only done when an
    external gateway is added. In case a new DVR+HA router was created
    without an external gateway, the l3 agent will raise an exception and
    will not create the router correctly. This patch adds a simple check to
    make sure that it is defined before it's actually used.
    
    Closes-Bug: #1560945
    
    (cherry picked from commit a8b60671150ac383c6ed24c26e773a97a476f7d2)
    
    Change-Id: I677e0837956a6d008a3935d961f078987a07d0c4

commit 4504a7431940443f036f9d1ab64e7255b2e11251
Author: Jakub Libosvar <libosvar@redhat.com>
Date:   Fri Mar 18 12:51:23 2016 +0000

conn_testers: Bump timeout for ICMPv6 echo tests
    
    In IPv6 scenarios NDP can increase round-trip time of ICMPv6 packets over
    1 seconds. The patch increases timeout for ICMPv6 to 2 seconds. Note
    that this will extend scenarios when ping is supposed to fail.
    
    Change-Id: Iec7d3138aee3fc904312dbc45ef76854ad0ea789
    Closes-Bug: 1557946
    (cherry picked from commit 420d5c7987f8ff7473d70697e5606f07ccfe7c1d)

commit 6d9774b0586d6b67e1af1c5a4bae2c55edcca852
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Mar 23 06:03:25 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I35b00d4e0b6f3429fd7173473c73e9dab4e6921e

commit a32f43d2b7d3d5ad319c45ffd8bfb55bcfaa6cd1
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Mar 22 06:03:08 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I81614a8ddfaf97a854964b0f1beab21432f110f6

commit 2b823de315aaca85bbafb020631f34b3fb233998
Author: Adelina Tuvenie <atuvenie@cloudbasesolutions.com>
Date:   Wed Sep 23 17:59:11 2015 -0700

Ovs agent can't start on Windows because of validate_local_ip
    
    Change I4b4527c28d0738890e33b343c9e17941e780bc24 introduced a
    validate_local_ip sanity check for the local_ip to see that it
    belongs to the host.
    
    This method uses linux specific implementation that fails on windows.
    
    This patch fixes this bug by adding a implementation for
    validate_local_ip that works on windows as well, using netifaces.
    
    Change-Id: Ia8299512687d9d7135fe013fbb38f2b28d54125d
    Closes-Bug: #1497940
    (cherry picked from commit 371e8aa0769086d069cc9005f1c454fb348afd46)

commit 3d0703c63d96a8448bbb2ef443f1754715ad00f5
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sun Mar 20 06:02:50 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I98ec552131ddd3848cce32a7150d378b31a8dbed

commit 165aa828b1cb63cb8be86ec081cea64aa8ada48c
Author: Akihiro Motoki <motoki@da.jp.nec.com>
Date:   Fri Mar 18 17:41:23 2016 +0900

Accept icmpv6 as protocol of SG rule for backward compatibility
    
    The patch https://review.openstack.org/#/c/252155/ renamed
    'icmpv6' protocol to 'ipv6-icmp'.
    This leads to backward compatiblity of the security group API.
    This commit allows to specify 'icmpv6' as well.
    
    TODO(amotoki): The constant for 'icmpv6' will be moved to
    neutron-lib soon after Mitaka is shipped.
    
    Change-Id: I0d7e1cd9fc075902449c5eb5ef27069083ab95d4
    Closes-Bug: #1558774
    (cherry picked from commit 85d638af455ae881ca45d2d390606ef1df5904b1)

commit f76056a94000857ac1613d08dee7858373964fa6
Author: Kevin Benton <kevin@benton.pub>
Date:   Wed Mar 16 12:28:49 2016 -0700

Outerjoin to networks for port ownership filter
    
    Change I55328cb43207654b9bb4cfb732923982d020ab0a
    added a port filter to compare tenant ID to the
    network owner as well. This caused the networks
    table to be added to the FROM statement since
    ports wasn't joined to networks for any other
    reason. This resulted in an explosion of records
    returned (networks * ports). SQLAlchemy would
    de-dup this for us when iterating over results;
    however, it would completely break the 'count()'
    operation required by get_ports_count (which
    the quota engine uses).
    
    Change-Id: I5b780121ba408fba691fff9304d4a22e5892b85f
    Closes-Bug: #1558827
    (cherry picked from commit ff4067af5ba52cc205f38d12cdf68bd454445ced)

commit 17a0d17b1ce943511962fb3c6760f847d63d44f2
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Mar 18 06:06:12 2016 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Idf9b605b5c2f4031d20510d4a9a2b7c9abf76788

commit af542a303bbdf261c2bf223c1d0a9472dd67538e
Author: Michael Johnson <johnsomor@gmail.com>
Date:   Thu Mar 17 06:31:42 2016 +0000

Update devstack plugin for dependent packages
    
    Recent changes to the gate base images [1] removed a package
    neutron requires (netcat-openbsd). This patch installs the
    required package.
    
    [1] https://review.openstack.org/#/c/292573
    
    Closes-bug: #1558397
    
    (cherry picked from commit 44ef44c0ff97d5b166d48d2ef93feafa9a0f7ea6)
    
    Change-Id: I4041478ca09bd124827782774b8520908ef07be0

commit e9088d09328793495aba260a05d6161f92aae4cd
Author: Armando Migliaccio <armamig@gmail.com>
Date:   Fri Mar 11 08:00:34 2016 -0800

Remove test_external_network_visibility
    
    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.
    
    [1] https://review.openstack.org/#/c/280427/
    
    Related-bug: 1553595
    Related-bug: 1552960
    
    (cherry-picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)
    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    Depends-on: I0219e17d2ec70939039344826d35cb533b3ce065

commit b682ec29301facb21d649e688b36541d8262552b
Author: Thierry Carrez <thierry@openstack.org>
Date:   Wed Mar 16 18:15:06 2016 +0100

Update .gitreview for stable/mitaka
    
    Change-Id: Iebba9a068678ee6861761a3c8cb22f64276442df

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-06-03: Fix included in openstack/neutron 9.0.0.0b1

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.