Quantum Network Manager - not clearing ips

Bug #988034 reported by Endre Karlson on 2012-04-24
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Somik Behera
Essex
Undecided
Unassigned

Bug Description

2012-04-24 20:09:19 ERROR nova.rpc.amqp [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-04-24 20:09:19 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line 556, in deallocate_for_instance
2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line 628, in update_dhcp
2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in kill_dhcp
2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid, run_as_root=True)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in _execute
2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd, **kwargs)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill -9 26037
2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command: kill -9 26037\n'
2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''

Related branches

Maru Newby (maru) wrote :

Am I missing something, or is this a nova bug? If so, you'll want to file it against nova rather than quantum.

Maru Newby (maru) wrote :

There also doesn't seem to be much in the way of information to aid in reproducing the problem. The sequence of events that led up to the bug would be helpful for whoever ends up fixing it.

I'm not sure how the traceback is related, but this may be similar to the
issue I ran into while giving the demo at the openstack conference :)

Brad, I think you were look at this, right?

Dan

On Tue, Apr 24, 2012 at 2:24 PM, Maru Newby <email address hidden> wrote:

> There also doesn't seem to be much in the way of information to aid in
> reproducing the problem. The sequence of events that led up to the bug
> would be helpful for whoever ends up fixing it.
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> New
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Brad Hall (bgh) wrote :

Review coming shortly..

Changed in quantum:
assignee: nobody → Brad Hall (bgh)
status: New → In Progress
importance: Undecided → High
Endre Karlson (endre-karlson) wrote :

It's taken from nova.network. It's a thing that occurs when using OVS.

Endre Karlson (endre-karlson) wrote :

Any news on this? It renders quantum kinda useless.

dan wendlandt (danwent) wrote :

Hi Endre, can you try out this patch and confirm it solves your issue: https://review.openstack.org/#/c/6949/1 ?

Endre Karlson (endre-karlson) wrote :
Download full text (13.3 KiB)

Doesn't seem to free up still when I destroy the instances.
So I go in and try to manually cleanup fixed ip's for the 2 networks:

+---------------------+---------------------+------------+---------+------+--------------+------------+-------------+-----------+--------+----------+----------------------+------+
| created_at | updated_at | deleted_at | deleted | id | address | network_id | instance_id | allocated | leased | reserved | virtual_interface_id | host |
+---------------------+---------------------+------------+---------+------+--------------+------------+-------------+-----------+--------+----------+----------------------+------+
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1297 | 172.16.65.16 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1298 | 172.16.65.17 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1299 | 172.16.65.18 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:08:38 | NULL | 0 | 1300 | 172.16.65.19 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:07:56 | NULL | 0 | 1301 | 172.16.65.20 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:11:05 | NULL | 0 | 1302 | 172.16.65.21 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:11:05 | NULL | 0 | 1303 | 172.16.65.22 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1304 | 172.16.65.23 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1305 | 172.16.65.24 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:48 | NULL | 0 | 1306 | 172.16.65.25 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1307 | 172.16.65.26 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1308 | 172.16.65.27 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:32:45 | NULL | 0 | 1309 | 172.16.65.28 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:32:44 | NULL | 0 | 1310 | 172.16.65.29 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-0...

Somik Behera (somikbehera) wrote :

Can you try this patch - https://review.openstack.org/#/c/6949/

and let us know if that resolves your issue? Also, in the above example, did you try the patch above and it still doesn't release your ips?

Endre Karlson (endre-karlson) wrote :

I tried the first patch yes, no positive feedback there.

2012/5/3 Somik Behera <email address hidden>

> Can you try this patch - https://review.openstack.org/#/c/6949/
>
> and let us know if that resolves your issue? Also, in the above example,
> did you try the patch above and it still doesn't release your ips?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Endre Karlson (endre-karlson) wrote :

Isn't that the same as before?
Den 1. mai 2012 03:50 skrev "dan wendlandt" <email address hidden>
følgende:

> Hi Endre, can you try out this patch and confirm it solves your issue:
> https://review.openstack.org/#/c/6949/1 ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Somik Behera (somikbehera) wrote :

The new patch has an additional fix to allow the default manager to clean up the ips in the DB as well, so its got an additional fix that was in my environment.

Endre Karlson (endre-karlson) wrote :

Yeah, after replacing my manager.py file it now seems to be ok.

Somik Behera (somikbehera) wrote :

Great! Glad to hear that. Did you just replace the manager.py from the patch link above or did you pull a certain version from the Quantum trunk?

Endre Karlson (endre-karlson) wrote :

I just replaced it. By the way can you take a look at my other bug reported
where it doesn't setup .conf files for dnsmasq?

2012/5/4 Somik Behera <email address hidden>

> Great! Glad to hear that. Did you just replace the manager.py from the
> patch link above or did you pull a certain version from the Quantum
> trunk?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Endre Karlson (endre-karlson) wrote :

Is there any news on this? Quantum is rendered useless at the moment.

Somik Behera (somikbehera) wrote :

Fix has been committed to Nova trunk - https://review.openstack.org/#/c/6949/

Changed in quantum:
assignee: Brad Hall (bgh) → Somik Behera (somikbehera)
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-05-18
affects: quantum → nova
Endre Karlson (endre-karlson) wrote :

I replaced my nova.network.quantum > manager and nova_ipam_lib files with the ones from upstream - restarted components and it works gloriously!

Devin Carlen (devcamcar) on 2012-05-22
Changed in nova:
milestone: none → folsom-1
Thierry Carrez (ttx) on 2012-05-23
Changed in nova:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/7970
Committed: http://github.com/openstack/nova/commit/3426030d3524e4e38c5843be0b9515805b67b04f
Submitter: Jenkins
Branch: stable/essex

commit 3426030d3524e4e38c5843be0b9515805b67b04f
Author: Somik Behera <email address hidden>
Date: Mon Apr 30 17:43:14 2012 -0700

    Fix bug 988034 - Quantum Network Manager - not clearing ips

    Added a line in nova_ipam_lib.QuantumNovaIPAMLib init
    method to ensure we clean up IPs and explanation of whats
    going on.

    Also, nuked release_fixed_ip as that is not
    needed in QuantumManager anymore.

    Change-Id: I3971b4d4c464ce68ba9370c9f5d8ca7840e0a72a

tags: added: in-stable-essex
Thierry Carrez (ttx) on 2012-09-27
Changed in nova:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers