OpenStack Compute (Nova)

Create/Destroy stress test bombs in NetworkManager.deallocate_fixed_ip

Reported by David Kranz on 2012-03-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Vish Ishaya

Bug Description

The tempest stress test that creates and destroys instances fails with the following in the network log:

2012-03-29 13:46:44 DEBUG nova.utils [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653b9f\
1c3eea65c011b] Running cmd (subprocess): cat /proc/6181/cmdline from (pid=1301) execute /usr/lib/python2.7/dist-packages/nova/utils\
.py:221
2012-03-29 13:46:44 DEBUG nova.utils [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653b9f\
1c3eea65c011b] Running cmd (subprocess): sudo nova-rootwrap kill -HUP 6181 from (pid=1301) execute /usr/lib/python2.7/dist-packages\
/nova/utils.py:221
2012-03-29 13:46:44 ERROR nova.rpc.amqp [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653\
b9f1c3eea65c011b] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 248, in deallocate_fixed_ip
(nova.rpc.amqp): TRACE: address)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1218, in deallocate_fixed_ip
(nova.rpc.amqp): TRACE: self.driver.release_dhcp(dev, address, vif['address'])
(nova.rpc.amqp): TRACE: TypeError: 'NoneType' object has no attribute '__getitem__'
(nova.rpc.amqp): TRACE:

Changed in nova:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Vish Ishaya (vishvananda)
David Kranz (david-kranz) wrote :

It is perhaps related but sometimes when I run this test I get this in the compute log, but with no traceback:

2012-03-29 15:14:30 ERROR nova.virt.libvirt.connection [-] Getting disk size of instance-00000030: [Errno 2] No such file or direct\
ory: '/var/lib/nova/instances/instance-00000030/disk'

Fix proposed to branch: master
Review: https://review.openstack.org/5968

Changed in nova:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/5968
Committed: http://github.com/openstack/nova/commit/be80a55d41f03a9e22d8d90a91bf6b9c09c0c692
Submitter: Jenkins
Branch: master

commit be80a55d41f03a9e22d8d90a91bf6b9c09c0c692
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Mar 29 11:44:34 2012 -0700

    Check vif exists before releasing ip

     * adds test to make sure code doesn't raise
     * fixes bug 968457

    Change-Id: I7110cb18a45fb955769247a9a0c5fb721ab3935a

Changed in nova:
status: In Progress → Fix Committed
David Kranz (david-kranz) wrote :

There may be another related problem with this. After installing RC2, I see these in the nova-network log:

2012-04-02 14:59:59 ERROR nova.rpc.amqp [req-3d2b44f3-456e-4475-93fc-1ad63ea47ad0 e5edfa94dbbc4cdc88ca50b31c4ee66b de85bc3386ff4c40\
9198228d302624f7] Exception during message handling
2012-04-02 14:59:59 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-04-02 14:59:59 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 248, in deallocate_\
fixed_ip
2012-04-02 14:59:59 TRACE nova.rpc.amqp address)
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1226, in deallocate_fixed_ip
2012-04-02 14:59:59 TRACE nova.rpc.amqp self.driver.release_dhcp(dev, address, vif['address'])
2012-04-02 14:59:59 TRACE nova.rpc.amqp TypeError: 'NoneType' object has no attribute '__getitem__'
2012-04-02 14:59:59 TRACE nova.rpc.amqp
2

Vish Ishaya (vishvananda) wrote :

looks like we missed the backport

Thierry Carrez (ttx) on 2012-04-03
tags: added: essex-rc-potential

Fix proposed to branch: master
Review: https://review.openstack.org/6170

Changed in nova:
status: Fix Committed → In Progress
Thierry Carrez (ttx) on 2012-04-03
Changed in nova:
milestone: none → essex-rc3
tags: removed: essex-rc-potential

Reviewed: https://review.openstack.org/6170
Committed: http://github.com/openstack/nova/commit/c96e75d6804d016da7c6356bf593eb86dcb2f257
Submitter: Jenkins
Branch: master

commit c96e75d6804d016da7c6356bf593eb86dcb2f257
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 10:07:05 2012 -0700

    Grab the vif directly on release instead of lookup

     * adds extra error handling and logging
     * safer fix for bug 968457

    Change-Id: I6d8c27c642e70dc701548550c0d94a8e0e64ce99

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/6107
Committed: http://github.com/openstack/nova/commit/256403e20fc8c5fb7ffcc68bc76ddfe39efb8cd9
Submitter: Jenkins
Branch: milestone-proposed

commit 256403e20fc8c5fb7ffcc68bc76ddfe39efb8cd9
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Mar 29 11:44:34 2012 -0700

    Check vif exists before releasing ip

     * adds test to make sure code doesn't raise
     * fixes bug 968457

    Change-Id: I7110cb18a45fb955769247a9a0c5fb721ab3935a

Changed in nova:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/6172
Committed: http://github.com/openstack/nova/commit/9adb6fbd809d6afda7aa637a2ed6a8436922dfea
Submitter: Jenkins
Branch: milestone-proposed

commit 9adb6fbd809d6afda7aa637a2ed6a8436922dfea
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 10:07:05 2012 -0700

    Grab the vif directly on release instead of lookup

     * adds extra error handling and logging
     * safer fix for bug 968457

    Change-Id: I6d8c27c642e70dc701548550c0d94a8e0e64ce99

Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc3 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers