neutron

Neutron deletes the router interface instead of adding a floatingip

Bug #1315467 reported by Attila Fazekas on 2014-05-02

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Attila Fazekas	neutron 2014.2 "juno"
	Icehouse	Fix Released	High	Attila Fazekas	neutron 2014.1.1

Bug Description

After parsing a lot of log files related to check failure, looks like the q-vpn at the time when I would expect to add
floating ip , it destroys the router's qg- and qr- interfaces.

However after the floating ip deletion request the q-vpn service restores the qg-, qr- interfaces.

tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,image,network,volume]
failed in http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/console.html.

admin user: admin/9e02f14321454af6bb27587770f27d9b
admin tenant id: admin/413bb1232bca45069f3a3256839effa1

test user: TestMinimumBasicScenario-1306090821/c8dd95056c0b407e8dd168dbf410a66a
test Tenant: TestMinimumBasicScenario-993819377/2527b8222e3343bca9f70343e608880c
External Net : public/c29040d3-7e73-4a87-9f73-bb5cbe602afb
External subnet: public-subnet/ee754eb6-6194-4a25-a4cc-f9233d366c1e
Network: TestMinimumBasicScenario-1375749858-network/f029d7a8-54e0-484c-a215-cc34066ae830
Subnet: TestMinimumBasicScenario-1375749858-subnet/7edc72be-1207-4571-95d4-911223885ae7 10.100.0.0/28
Router id:TestMinimumBasicScenario-1375749858-router/08216822-5ee2-4313-be7e-dad2d84147db
Expected interfaces in the qrouter-08216822-5ee2-4313-be7e-dad2d84147db:
* lo 127.0.0.1
* qr-529eddd4-2c 10.100.0.1/28 iface_id: 529eddd4-2ca8-43ec-9cab-29c3a6632604, attached-mac: fa:16:3e:2a:f8:ba, ofport 166
* qg-9be8f502-93 172.24.4.85/24 iface-id: 9be8f502-9360-47dc-9eff-33c8743e7c2b attached-mac: fa:16:3e:be:a1:54, ofport 37

Floating IP: 172.24.4.87 (Never appears in the q-vpn log)
port: (net/subnet/port)(c29040d3-7e73-4a87-9f73-bb5cbe602afb/ee754eb6-6194-4a25-a4cc-f9233d366c1e/013b0b2d-80ed-403d-b380-6b6895ce34f5)
mac?: fa:16:3e:15:f2:57
floating ip uuid: cd84111e-af6a-4c26-af73-3167419c664a

Instance:
Ipv4: 10.100.0.2 mac: FA:16:3E:18:F1:69
Instance uuid: 8a552eda-2fbd-4972-bfcf-cee7e6472871
iface_id/port_id: 4188c532-3265-4294-8b4e-9bbfe5a482e8
ovsdb_interface_uuid: 9d7b858b-745e-482b-b91a-1e9ae34fc545
intbr tag: 49

dhcp server dev: tap4c6c6e06-e4
ns: qdhcp-f029d7a8-54e0-484c-a215-cc34066ae830
ip: 10.100.0.3 mac: fa:16:3e:a2:f1:ea
intbr tag: 49

Host:
eth0: 10.7.16.229/15 mac: 02:16:3e:52:5d:ff

Router + router interface creation in the logs:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_49_46_924
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-svc.txt.gz#_2014-05-01_19_49_46_724

Floating IP create:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_447
Floating IP associate:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_814

q-vpn starts destroying the router:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_50_20_277
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-c0cd93e7-5cb0-403b-a509-8e07b352b89d', 'ipsec', 'whack', '--ctlbase', '/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto', '--status']
Exit code: 1
Stdout: ''
Stderr: 'whack: Pluto is not running (no "/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto.ctl")\n' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.277 19279 DEBUG neutron.openstack.common.lockutils [-] Semaphore / lock released "sync" inner /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:252
2014-05-01 19:50:20.277 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:20.516 19279 DEBUG neutron.agent.linux.utils [-]
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c']
Exit code: 0
Stdout: '605: qr-529eddd4-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:2a:f8:ba brd ff:ff:ff:ff:ff:ff\n'
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.517 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-int'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:20.533 19279 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', '-o', 'link', 'show', 'br-int']
Exit code: 0
Stdout: '6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN \\ link/ether c6:49:9f:72:d2:4a brd ff:ff:ff:ff:ff:ff\n'
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.534 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:21.061 19279 DEBUG neutron.agent.linux.utils [-]
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c']
Exit code: 0

Tempest tries to connect to the VM:
2014-05-01 19:50:19,463 .. 2014-05-01 19:53:48,494 ~ 209 sec

Delete request for the floatingip:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_24_191
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_54_25_047

Neutron touches the qr-529eddd4-2c interface + port,
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_54_37_886

Delete request for router and router interface:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_57_423

See original description

Revision history for this message

Attila Fazekas (afazekas) wrote on 2014-05-02:

bug_report_in_txt Edit (6.0 KiB, text/plain)

May be it is better readable as a txt.

Attila Fazekas (afazekas) on 2014-05-03

description:

updated

Revision history for this message

Attila Fazekas (afazekas) wrote on 2014-05-04:

Review: https://review.openstack.org/#/c/92018/

Changed in neutron:
assignee:	nobody → Attila Fazekas (afazekas)
status:	New → In Progress

Kyle Mestery (mestery) on 2014-05-04

Changed in neutron:
importance:	Undecided → High

Attila Fazekas (afazekas) on 2014-05-05

tags:

added: icehouse-backport-potential

Revision history for this message

Openstack Gerrit (openstack-gerrit) wrote on 2014-05-05: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/92147

Revision history for this message

Openstack Gerrit (openstack-gerrit) wrote on 2014-05-07: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/92018
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=45381fe1c742c75773d97f1c0bd1f3cb1e7a6468
Submitter: Jenkins
Branch: master

commit 45381fe1c742c75773d97f1c0bd1f3cb1e7a6468
Author: Attila Fazekas <email address hidden>
Date: Sun May 4 19:54:37 2014 +0200

L3 RPC loop could delete a router on concurrent update

routers_updated does not acquire any lock just updates
a set for future rpc loop processing.

    The self.updated_routers can be changed by concurrent update
    notification. If this change happens at the time around the
    self.plugin_rpc.get_routers call, the additional routers
    - by mistake - is considered as admin_state_up=false routers, which
     are safe to delete.

    Creating a local copy of the updated_routers and preserve
    the fresh updated_routers entries for the next _rpc_loop
    operations.

    Change-Id: Icc7377f9c29e248c3b34562465e859b15ecc2ec3
    Closes-Bug: #1315467
    Partial-Bug: #1253896

Changed in neutron:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-07: Fix proposed to neutron (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/92630

Kyle Mestery (mestery) on 2014-05-08

Changed in neutron:
milestone:	none → juno-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-28: Fix merged to neutron (stable/icehouse)

Reviewed: https://review.openstack.org/92630
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=74a9365edd9fb2206c52d8ee9fa781b772d1738d
Submitter: Jenkins
Branch: stable/icehouse

commit 74a9365edd9fb2206c52d8ee9fa781b772d1738d
Author: Attila Fazekas <email address hidden>
Date: Sun May 4 19:54:37 2014 +0200

L3 RPC loop could delete a router on concurrent update

routers_updated does not acquire any lock just updates
a set for future rpc loop processing.

    Creating a local copy of the updated_routers and preserve
    the fresh updated_routers entries for the next _rpc_loop
    operations.

    Change-Id: Icc7377f9c29e248c3b34562465e859b15ecc2ec3
    Closes-Bug: #1315467
    Partial-Bug: #1253896
    (cherry picked from commit 45381fe1c742c75773d97f1c0bd1f3cb1e7a6468)