resize-revert might be failed when using slow NFS
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Guangya Liu (Jay Lau) |
Bug Description
I did the followig steps with a script.
1) resize vm
2) resize-revert vm
3) goto 1)
Sometimes found resize failed with following error:
[root@rhel8233 ~]# nova show vm1
+------
| Property | Value |
+------
| status | ERROR |
| updated | 2013-07-
| OS-EXT-
| OS-EXT-
| key_name | None |
| image | cirros-0.3.0-x86_64 (9e66c9bb-
| hostId | 0ddc9ca22d1dece
| OS-EXT-STS:vm_state | error |
| OS-EXT-
| OS-SRV-
| OS-EXT-
| flavor | m1.tiny (1) |
| id | fb33fb20-
| security_groups | [{u'name': u'default'}] |
| OS-SRV-
| user_id | e18753eee1ec423
| name | vm1 |
| created | 2013-07-
| tenant_id | 333416515ef44d5
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| fault | {u'message': u'OSError', u'code': 500, u'details': u'[Errno 2] No such file or directory: \'/var/
| | File "/usr/lib/
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/
| | block_device_info, power_on) |
| | File "/usr/lib/
| | self._cleanup_
| | File "/usr/lib/
| | shutil.
| | File "/usr/lib64/
| | onerror(os.listdir, path, sys.exc_info()) |
| | File "/usr/lib64/
| | names = os.listdir(path) |
| | ', u'created': u'2013-
| OS-EXT-
| OS-EXT-
| config_drive | |
+------
After some debug, found that in driver.py:: finish_
def finish_
inst_base = libvirt_
# NOTE(danms): if we're recovering from a failed migration,
# make sure we don't have a left-over same-host base directory
# that would conflict. Also, don't fail on the rename if the
# failure happened early.
if os.path.
if os.path.
Changed in nova: | |
status: | New → In Progress |
tags: | added: libvirt |
Changed in nova: | |
status: | In Progress → Fix Committed |
Changed in nova: | |
milestone: | none → havana-rc1 |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | havana-rc1 → 2013.2 |
Check nova compute log:
resize target host report the VM was destroyed in 23:56:18 libvirt. driver [req-c7d7450a- e665-416f- a392-9b760f123f c9 2785a0d1141c450 2b573155988382a 27 1394848571b8408 188a239da2a1bb3 51] [instance: 7bccaf62- ea4d-4e27- b4ff-d715fa2f9e 9d] Deleting instance files /var/lib/ nova/instances/ 7bccaf62- ea4d-4e27- b4ff-d715fa2f9e 9d
2013-07-29 23:56:18.260 8105 INFO nova.virt.
But on source host of the VM, it can still detect that the os.path. exists( inst_base) is true. (inst_base: /var/lib/ nova/instances/ 7bccaf62- ea4d-4e27- b4ff-d715fa2f9e 9d) manager [req-c7d7450a- e665-416f- a392-9b760f123f c9 2785a0d1141c450 2b573155988382a 27 1394848571b8408 188a239da2a1bb3 51] [instance: 7bccaf62- ea4d-4e27- b4ff-d715fa2f9e 9d] [Errno 2] No such file or directory: '/var/lib/ nova/instances/ 7bccaf62- ea4d-4e27- b4ff-d715fa2f9e 9d'. Setting instance vm_state to ERROR common. rpc.amqp [req-c7d7450a- e665-416f- a392-9b760f123f c9 2785a0d1141c450 2b573155988382a 27 1394848571b8408 188a239da2a1bb3 51] Exception during message handling common. rpc.amqp Traceback (most recent call last): common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/openstack/ common/ rpc/amqp. py", line 421, in _process_data common. rpc.amqp **args) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/openstack/ common/ rpc/dispatcher. py", line 172, in dispatch common. rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/exception. py", line 100, in wrapped common. rpc.amqp temp_level, payload) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/exception. py", line 77, in wrapped common. rpc.amqp return f(self, context, *args, **kw) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 235, in decorated_function common. rpc.amqp pass common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 221, in decorated_function common. rpc.amqp return function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 286, in decorated_function common. rpc.amqp function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 263, in decor...
2013-07-29 23:56:19.704 8365 ERROR nova.compute.
2013-07-29 23:56:20.003 8365 ERROR nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.
2013-07-29 23:56:20.003 8365 TRACE nova.openstack.