Create/Destroy stress test bombs in NetworkManager.deallocate_fixed_ip

[David Kranz]

The tempest stress test that creates and destroys instances fails with the following in the network log:

2012-03-29 13:46:44 DEBUG nova.utils [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653b9f\
1c3eea65c011b] Running cmd (subprocess): cat /proc/6181/cmdline from (pid=1301) execute /usr/lib/python2.7/dist-packages/nova/utils\
.py:221
2012-03-29 13:46:44 DEBUG nova.utils [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653b9f\
1c3eea65c011b] Running cmd (subprocess): sudo nova-rootwrap kill -HUP 6181 from (pid=1301) execute /usr/lib/python2.7/dist-packages\
/nova/utils.py:221
2012-03-29 13:46:44 ERROR nova.rpc.amqp [req-d787a30a-5c39-4272-8177-973fa1f37856 034946ec763547ef872c0a87291748b6 ec2e09808a2e4653\
b9f1c3eea65c011b] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 248, in deallocate_fixed_ip
(nova.rpc.amqp): TRACE: address)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1218, in deallocate_fixed_ip
(nova.rpc.amqp): TRACE: self.driver.release_dhcp(dev, address, vif['address'])
(nova.rpc.amqp): TRACE: TypeError: 'NoneType' object has no attribute '__getitem__'
(nova.rpc.amqp): TRACE:

[David Kranz (david-kranz)]

It is perhaps related but sometimes when I run this test I get this in the compute log, but with no traceback:

2012-03-29 15:14:30 ERROR nova.virt.libvirt.connection [-] Getting disk size of instance-00000030: [Errno 2] No such file or direct\
ory: '/var/lib/nova/instances/instance-00000030/disk'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5968

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5968
Committed: http://github.com/openstack/nova/commit/be80a55d41f03a9e22d8d90a91bf6b9c09c0c692
Submitter: Jenkins
Branch: master

commit be80a55d41f03a9e22d8d90a91bf6b9c09c0c692
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Mar 29 11:44:34 2012 -0700

    Check vif exists before releasing ip

     * adds test to make sure code doesn't raise
     * fixes bug 968457

    Change-Id: I7110cb18a45fb955769247a9a0c5fb721ab3935a

[David Kranz (david-kranz)]

There may be another related problem with this. After installing RC2, I see these in the nova-network log:

2012-04-02 14:59:59 ERROR nova.rpc.amqp [req-3d2b44f3-456e-4475-93fc-1ad63ea47ad0 e5edfa94dbbc4cdc88ca50b31c4ee66b de85bc3386ff4c40\
9198228d302624f7] Exception during message handling
2012-04-02 14:59:59 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-04-02 14:59:59 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 248, in deallocate_\
fixed_ip
2012-04-02 14:59:59 TRACE nova.rpc.amqp address)
2012-04-02 14:59:59 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1226, in deallocate_fixed_ip
2012-04-02 14:59:59 TRACE nova.rpc.amqp self.driver.release_dhcp(dev, address, vif['address'])
2012-04-02 14:59:59 TRACE nova.rpc.amqp TypeError: 'NoneType' object has no attribute '__getitem__'
2012-04-02 14:59:59 TRACE nova.rpc.amqp
2

[Vish Ishaya (vishvananda)]

looks like we missed the backport

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/6170

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6170
Committed: http://github.com/openstack/nova/commit/c96e75d6804d016da7c6356bf593eb86dcb2f257
Submitter: Jenkins
Branch: master

commit c96e75d6804d016da7c6356bf593eb86dcb2f257
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 10:07:05 2012 -0700

    Grab the vif directly on release instead of lookup

     * adds extra error handling and logging
     * safer fix for bug 968457

    Change-Id: I6d8c27c642e70dc701548550c0d94a8e0e64ce99

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6107
Committed: http://github.com/openstack/nova/commit/256403e20fc8c5fb7ffcc68bc76ddfe39efb8cd9
Submitter: Jenkins
Branch: milestone-proposed

commit 256403e20fc8c5fb7ffcc68bc76ddfe39efb8cd9
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Mar 29 11:44:34 2012 -0700

    Check vif exists before releasing ip

     * adds test to make sure code doesn't raise
     * fixes bug 968457

    Change-Id: I7110cb18a45fb955769247a9a0c5fb721ab3935a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6172
Committed: http://github.com/openstack/nova/commit/9adb6fbd809d6afda7aa637a2ed6a8436922dfea
Submitter: Jenkins
Branch: milestone-proposed

commit 9adb6fbd809d6afda7aa637a2ed6a8436922dfea
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Apr 3 10:07:05 2012 -0700

    Grab the vif directly on release instead of lookup

     * adds extra error handling and logging
     * safer fix for bug 968457

    Change-Id: I6d8c27c642e70dc701548550c0d94a8e0e64ce99

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/177450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177450
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a7521522eefe11ebbe3b39a851d7c7144cc50a53
Submitter: Jenkins
Branch: master

commit a7521522eefe11ebbe3b39a851d7c7144cc50a53
Author: Matt Riedemann <email address hidden>
Date: Fri Apr 24 13:53:24 2015 -0700

    n-net: turn down log level when vif isn't found in deallocate_fixed_ip

    Logstash shows that within 7 days this error shows up over 16,000 times
    and 85% of those jobs are successful.

    http://goo.gl/mz8ocj

    Given bug 968457 was reported in 2012 and nothing has been adjusted
    since to address this race, we can assume it's not an error condition
    that justifies the ERROR log level, so let's tone it down to INFO.

    Related-Bug: #968457

    Change-Id: I050a8915349da39530433edaba49c1e781e6d692