OpenStack Compute (nova)

nova instance delete fails if dhcp_release fails

Bug #1289397 reported by satyadev svn on 2014-03-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Low	Gary Kotton	OpenStack Compute (nova) 2014.2 "juno"

Bug Description

ssatya@devstack:~$ nova boot --image 1e95fe6b-cec6-4420-97d1-1e7bc8c81c49 --flavor 1 testdummay
+--------------------------------------+-----------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | networking |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | fK8SPGtHLUds |
| config_drive | |
| created | 2014-03-07T14:33:49Z |
| flavor | m1.tiny (1) |
| hostId | 2c1ae30aa2a235d9c0c8b04aae3f4199cd98356e44a03b5c8f878adb |
| id | eae503d9-c6f7-4e3e-9adc-0b8b6803c90e |
| image | debian-2.6.32-i686 (1e95fe6b-cec6-4420-97d1-1e7bc8c81c49) |
| key_name | - |
| metadata | {} |
| name | testdummay |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 209ab7e4f3744675924212805db3ad74 |
| updated | 2014-03-07T14:33:50Z |
| user_id | f3756a4910054883b84ee15acc15fbd1 |
+--------------------------------------+-----------------------------------------------------------+
ssatya@devstack:~$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| eae503d9-c6f7-4e3e-9adc-0b8b6803c90e | testdummay | BUILD | spawning | NOSTATE | |
| d1e982c4-85c2-422d-b046-1643bd81e674 | testvm1 | ERROR | deleting | Shutdown | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+
ssatya@devstack:~$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| eae503d9-c6f7-4e3e-9adc-0b8b6803c90e | testdummay | ACTIVE | - | Running | private=10.0.0.3 |
| d1e982c4-85c2-422d-b046-1643bd81e674 | testvm1 | ERROR | deleting | Shutdown | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+
ssatya@devstack:~$ nova stop testdummay
ssatya@devstack:~$ nova list
+--------------------------------------+------------+---------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+---------+------------+-------------+------------------+
| eae503d9-c6f7-4e3e-9adc-0b8b6803c90e | testdummay | SHUTOFF | - | Shutdown | private=10.0.0.3 |
| d1e982c4-85c2-422d-b046-1643bd81e674 | testvm1 | ERROR | deleting | Shutdown | private=10.0.0.2 |
+--------------------------------------+------------+---------+------------+-------------+------------------+
ssatya@devstack:~$ nova delete testdummay
ssatya@devstack:~$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| eae503d9-c6f7-4e3e-9adc-0b8b6803c90e | testdummay | ERROR | deleting | Shutdown | private=10.0.0.3 |
| d1e982c4-85c2-422d-b046-1643bd81e674 | testvm1 | ERROR | deleting | Shutdown | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+

Tags:

Vipin Balachandran (vbala) on 2014-03-10

Changed in cinder:
status:	New → Incomplete

Revision history for this message

Vipin Balachandran (vbala) wrote on 2014-03-10:

Hi Satya,

I am not getting how this is related to the vmdk driver.
Please provide some more details like exceptions or error messages.

Thanks,
Vipin

Changed in cinder:
assignee:	nobody → Vipin Balachandran (vbala)

Revision history for this message

satyadev svn (svnsatya) wrote on 2014-03-10:

issue is not related to vmdk. issue related to nova only.

Vipin Balachandran (vbala) on 2014-03-10

Changed in cinder:
assignee:	Vipin Balachandran (vbala) → nobody
affects:	cinder → nova

Tracy Jones (tjones-i) on 2014-03-12

Changed in nova:
assignee:	nobody → Tracy Jones (tjones-i)

Revision history for this message

Tracy Jones (tjones-i) wrote on 2014-03-12:

Download full text (6.2 KiB)

I was not able to repro this. Please try again and attach logs if you have failures

I was not able to repro this.   Please try again and attach logs if you have failures

[~/devstack] ➔ glance image-list
+--------------------------------------+--------------------+-------------+------------------+-----------+--------+
| ID                                   | Name               | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------------------+-------------+------------------+-----------+--------+
| 96ede0b8-8a22-4c67-aba7-f7efa3828818 | debian-2.6.32-i686 | vmdk        | bare             | 298288412 | active |
+--------------------------------------+--------------------+-------------+------------------+-----------+--------+
[~/devstack] ➔ nova boot --image debian-2.6.32-i686  --flavor 1 testdummy
+--------------------------------------+-----------------------------------------------------------+
| Property                             | Value                                                     |
+--------------------------------------+-----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                    |
| OS-EXT-AZ:availability_zone          | nova                                                      |
| OS-EXT-STS:power_state               | 0                                                         |
| OS-EXT-STS:task_state                | scheduling                                                |
| OS-EXT-STS:vm_state                  | building                                                  |
| OS-SRV-USG:launched_at               | -                                                         |
| OS-SRV-USG:terminated_at             | -                                                         |
| accessIPv4                           |                                                           |
| accessIPv6                           |                                                           |
| adminPass                            | DXf6C2dN5d96                                              |
| config_drive                         |                                                           |
| created                              | 2014-03-12T20:40:37Z                                      |
| flavor                               | m1.tiny (1)                                               |
| hostId                               |                                                           |
| id                                   | 074877ac-ea17-4281-87ca-5cad987a3c2f                      |
| image                                | debian-2.6.32-i686 (96ede0b8-8a22-4c67-aba7-f7efa3828818) |
| key_name                             | -                                                         |
| metadata                             | {}                                                        |
| name                                 | testdummy                                                 |
| os-extended-volumes:volumes_attached | []                                                        |
| progress                             | 0                                                         |
| security_groups                      | default                                                   |
| status                               | BUILD                                                     |
| tenant_id                            | d8ee70c6605b4c9a88beba13e6f16cba                          |
| updated                              | 2014-03-12T20:40:38Z                                      |
| user_id                              | 342809bc4309432fabba583d016fcbed                          |
+--------------------------------------+-----------------------------------------------------------+
[~/devstack] ➔ nova list
+--------------------------------------+-----------+--------+------------+-------------+----------+
| ID                                   | Name      | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------+
| 074877ac-ea17-4281-87ca-5cad987a3c2f | testdummy | BUILD  | spawning   | NOSTATE     |          |
+--------------------------------------+-----------+--------+------------+-------------+----------+
[~/devstack] ➔ nova list
+--------------------------------------+-----------+--------+------------+-------------+----------+
| ID                                   | Name      | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------+
| 074877ac-ea17-4281-87ca-5cad987a3c2f | testdummy | BUILD  | spawning   | NOSTATE     |          |
+--------------------------------------+-----------+--------+------------+-------------+----------+
[~/devstack] ➔ nova list
+--------------------------------------+-----------+--------+------------+-------------+------------------+
| ID                                   | Name      | Status | Task State | Power State | Networks         |
+--------------------------------------+-----------+--------+------------+-------------+------------------+
| 074877ac-ea17-4281-87ca-5cad987a3c2f | testdummy | ACTIVE | -          | Running     | private=10.0.0.2 |
+--------------------------------------+-----------+--------+------------+-------------+------------------+
[~/devstack] ➔ nova stop testdummy
[~/devstack] ➔ nova list
+--------------------------------------+-----------+---------+------------+-------------+------------------+
| ID                                   | Name      | Status  | Task State | Power State | Networks         |
+--------------------------------------+-----------+---------+------------+-------------+------------------+
| 074877ac-ea17-4281-87ca-5cad987a3c2f | testdummy | SHUTOFF | -          | Shutdown    | private=10.0.0.2 |
+--------------------------------------+-----------+---------+------------+-------------+------------------+
[~/devstack] ➔ nova delete testdummy
[~/devstack] ➔ 
[~/devstack] ➔ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[~/devstack] ➔

Revision history for this message

satyadev svn (svnsatya) wrote on 2014-03-13:

screen-n-cpu.2014-03-12-121508.log Edit (296.4 KiB, text/plain)

Please try with demo user. See attached log for reference

satyadev svn (svnsatya) on 2014-03-14

Changed in nova:
status:	Incomplete → In Progress
status:	In Progress → Invalid

Tracy Jones (tjones-i) on 2014-03-14

Changed in nova:
assignee:	Tracy Jones (tjones-i) → nobody

satyadev svn (svnsatya) on 2014-04-07

Changed in nova:
status:	Invalid → New

Tracy Jones (tjones-i) on 2014-04-23

Changed in nova:
importance:	Undecided → Low

Revision history for this message

Gary Kotton (garyk) wrote on 2014-06-05:

Hi,
This looks like a networking issue:

[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m timeout=timeout)
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 390, in send
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m return self._send(target, ctxt, message, wait_for_reply, timeout)
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 383, in _send
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m raise result
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mRemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.3 fa:16:3e:62:ab:ae
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mExit code: 1
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mStdout: ''
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mStderr: 'cannot setup interface: No such device\n'
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m[u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch\n incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n return self._dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/opt/stack/nova/nova/network/manager.py", line 564, in deallocate_for_instance\n instance=instance)\n', u' File "/opt/stack/nova/nova/network/manager.py", line 1526, in deallocate_fixed_ip\n instance=instance)\n', u' File "/opt/stack/nova/nova/network/manager.py", line 963, in deallocate_fixed_ip\n self.driver.release_dhcp(dev, address, vif.address)\n', u' File "/opt/stack/nova/nova/network/linux_net.py", line 993, in release_dhcp\n utils.execute(\'dhcp_release\', dev, address, mac_address, run_as_root=True)\n', u' File "/opt/stack/nova/nova/utils.py", line 164, in execute\n return processutils.execute(*cmd, **kwargs)\n', u' File "/opt/stack/nova/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n', u"ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.3 fa:16:3e:62:ab:ae\nExit code: 1\nStdout: ''\nStderr: 'cannot setup interface: No such device\\n'\n"].

It is not related to the vmware driver.
Thanks
Gary

Hi,
This looks like a networking issue:

[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m    timeout=timeout)
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 390, in send
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m    return self._send(target, ctxt, message, wait_for_reply, timeout)
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 383, in _send
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m    raise result
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mRemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.3 fa:16:3e:62:ab:ae
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mExit code: 1
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mStdout: ''
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00mStderr: 'cannot setup interface: No such device\n'
[01;31m2014-03-13 16:47:01.517 TRACE oslo.messaging._executors.base [01;35m[00m[u'Traceback (most recent call last):\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch\n    incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n    return self._dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/opt/stack/nova/nova/network/manager.py", line 564, in deallocate_for_instance\n    instance=instance)\n', u'  File "/opt/stack/nova/nova/network/manager.py", line 1526, in deallocate_fixed_ip\n    instance=instance)\n', u'  File "/opt/stack/nova/nova/network/manager.py", line 963, in deallocate_fixed_ip\n    self.driver.release_dhcp(dev, address, vif.address)\n', u'  File "/opt/stack/nova/nova/network/linux_net.py", line 993, in release_dhcp\n    utils.execute(\'dhcp_release\', dev, address, mac_address, run_as_root=True)\n', u'  File "/opt/stack/nova/nova/utils.py", line 164, in execute\n    return processutils.execute(*cmd, **kwargs)\n', u'  File "/opt/stack/nova/nova/openstack/common/processutils.py", line 193, in execute\n    cmd=\' \'.join(cmd))\n', u"ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.3 fa:16:3e:62:ab:ae\nExit code: 1\nStdout: ''\nStderr: 'cannot setup interface: No such device\\n'\n"].

It is not related to the vmware driver.
Thanks
Gary

tags:

added: network
removed: nova vmware

Revision history for this message

Gary Kotton (garyk) wrote on 2014-06-05:

https://review.openstack.org/98060

Changed in nova:
assignee:	nobody → Gary Kotton (garyk)
milestone:	none → juno-1

Revision history for this message

John Garbutt (johngarbutt) wrote on 2014-06-10:

We use milestones only at the very end of the milestone to track bugs that should block the release of the milestone. Please use tags not milestones to track bugs, try things like: my_bug_topic_milestone_name

Changed in nova:
milestone:	juno-1 → none

Tracy Jones (tjones-i) on 2014-06-18

Changed in nova:
status:	New → Triaged

OpenStack Infra (hudson-openstack) on 2014-06-24

Changed in nova:
status:	Triaged → In Progress

Sean Dague (sdague) on 2014-09-17

summary:

- vmware: nova instance delete - show status error
+ nova instance delete fails if dhcp_release fails

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-18: Fix merged to nova (master)

Reviewed: https://review.openstack.org/98060
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a516ae71d57eda013f7cb9428f945dfea08ead3e
Submitter: Jenkins
Branch: master

commit a516ae71d57eda013f7cb9428f945dfea08ead3e
Author: Gary Kotton <email address hidden>
Date: Thu Jun 5 01:35:32 2014 -0700

Network: enable instance deletion when dhcp release fails

In the event that the 'dhcp_release' fails we should continue
with the deletion of the instance.

This will no longer the leave the instance in ERROR state.

Change-Id: Ie8259180a9df12865940907c728a8d9da3c9fda0
Closes-bug: #1289397

Changed in nova:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-10-01

Changed in nova:
milestone:	none → juno-rc1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in nova:
milestone:	juno-rc1 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

screen-n-cpu.2014-03-12-121508.log Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.