OpenStack Compute (nova)

intance boots failed because of injecting a file to a not exist path in image

Bug #1272829 reported by wangpan on 2014-01-26

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Undecided	wangpan	OpenStack Compute (nova) 2014.1 "icehouse"

Bug Description

reproduce steps in my devstack env with latest master branch:
1. change the nova.conf (#force_config_drive = always) and restart nova-compute
2. download the cirros bootable qcow2 image cirros-0.3.0-x86_64-disk.img(wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img)
3. upload the image to glance
4. nova boot --flavor m1.nano --image cirros-0.3.0-x86_64-disk.img --file /etc/xxx/yyy=./start_nvs.sh test-inject

the reason is that, we don't `mkdir -p /etc/xxx/yyy` before `tee` like the folsom version.

The trace log:
2014-01-26 11:59:18.948 ERROR nova.compute.manager [req-91698f94-67b6-483e-a6cf-1cade5f19fe6 admin admin] [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Error: Unexpected error while running command.
Command: tee /tmp/openstack-vfs-localfsrGqEY7/etc/xxx/yyy
Exit code: 1
Stdout: '#! /bin/bash\nsudo pkill keystone\nsudo pkill glance\nsudo pkill nova-\n\nsudo keystone-all > ~/key.log 2>&1 &\nsudo glance-api > ~/gapi.log 2>&1 &\nsudo glance-registry > ~/greg.log 2>&1 &\n\nsudo nova-conductor >~/cond.log 2>&1 &\nsudo nova-api >~/api.log 2>&1 &\nsudo nova-scheduler >~/sch.log 2>&1 &\nsudo nova-compute >~/com.log 2>&1 &\nsudo nova-network >~/net.log 2>&1 &\n'
Stderr: 'tee: /tmp/openstack-vfs-localfsrGqEY7/etc/xxx/yyy: No such file or directory\n'
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Traceback (most recent call last):
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/compute/manager.py", line 1054, in _build_instance
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] set_access_ip=set_access_ip)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] return function(self, context, *args, **kwargs)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/compute/manager.py", line 1463, in _spawn
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] six.reraise(self.type_, self.value, self.tb)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/compute/manager.py", line 1460, in _spawn
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] block_device_info)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2182, in spawn
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] admin_pass=admin_password)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2592, in _create_image
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] injection_path = image('disk').path
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] six.reraise(self.type_, self.value, self.tb)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2592, in _create_image
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] injection_path = image('disk').path
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/disk/api.py", line 351, in inject_data
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] admin_password, files, mandatory)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/disk/api.py", line 431, in inject_data_into_fs
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] inject_func(inject_val, fs)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/disk/api.py", line 443, in _inject_files_into_fs
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] _inject_file_into_fs(fs, path, contents)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/disk/api.py", line 452, in _inject_file_into_fs
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] fs.replace_file(path, contents)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/virt/disk/vfs/localfs.py", line 123, in replace_file
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] utils.execute('tee', *args, **kwargs)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/utils.py", line 166, in execute
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] return processutils.execute(*cmd, **kwargs)
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] File "/opt/stack/nova/nova/openstack/common/processutils.py", line 178, in execute
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] cmd=' '.join(cmd))
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] ProcessExecutionError: Unexpected error while running command.
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Command: tee /tmp/openstack-vfs-localfsrGqEY7/etc/xxx/yyy
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Exit code: 1
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Stdout: '#! /bin/bash\nsudo pkill keystone\nsudo pkill glance\nsudo pkill nova-\n\nsudo keystone-all > ~/key.log 2>&1 &\nsudo glance-api > ~/gapi.log 2>&1 &\nsudo glance-registry > ~/greg.log 2>&1 &\n\nsudo nova-conductor >~/cond.log 2>&1 &\nsudo nova-api >~/api.log 2>&1 &\nsudo nova-scheduler >~/sch.log 2>&1 &\nsudo nova-compute >~/com.log 2>&1 &\nsudo nova-network >~/net.log 2>&1 &\n'
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e] Stderr: 'tee: /tmp/openstack-vfs-localfsrGqEY7/etc/xxx/yyy: No such file or directory\n'
2014-01-26 11:59:18.948 TRACE nova.compute.manager [instance: aa481bea-2822-4eca-89b1-d0d7734d9b9e]

See original description

Tags:

wangpan (hzwangpan) on 2014-01-26

description:

updated

Sahid Orentino (sahid-ferdjaoui) on 2014-01-26

Changed in nova:
status:	New → Confirmed
tags:	added: compute

Revision history for this message

Sahid Orentino (sahid-ferdjaoui) wrote on 2014-01-26:

I have reproduced the bug with libvirt driver.

Changed in nova:
assignee:	nobody → sahid (sahid-ferdjaoui)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-26: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/69185

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

wangpan (hzwangpan) wrote on 2014-01-27:

Hi sahid, I have already commited a patch here:
https://review.openstack.org/#/c/69175/

wangpan (hzwangpan) on 2014-01-28

Changed in nova:
assignee:	sahid (sahid-ferdjaoui) → wangpan (hzwangpan)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-03: Fix merged to nova (master)

Reviewed: https://review.openstack.org/69175
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=af8972a21b1f6da7ef470bae52fdbd47c3a5afeb
Submitter: Jenkins
Branch: master

commit af8972a21b1f6da7ef470bae52fdbd47c3a5afeb
Author: Wangpan <email address hidden>
Date: Sun Jan 26 14:38:00 2014 +0800

Ensure parent dir exists while injecting files

    Currently, if we inject a file to a not existing path,
    the injecting will be failed with error 'No such file
    or directory'.
    Nova will firstly execute `mkdir -p` in folsom before injecting,
    but this operation is lost from commit
    72918415f2b6a241d6097a563aba55a849936591

This patch fix this bug by adding the `mkdir -p` operation as folsom.

Closes-bug: #1272829

Change-Id: If33a79c538e07cb25300db64ac7a9dee3d3b19c2

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

wangpan (hzwangpan) wrote on 2014-03-05:

This bug also exists in stable havana, we should backport this commit to havana.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-05: Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/78074

Thierry Carrez (ttx) on 2014-03-05

Changed in nova:
milestone:	none → icehouse-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-04-17

Changed in nova:
milestone:	icehouse-3 → 2014.1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-31: Change abandoned on nova (stable/havana)

Change abandoned by Wangpan (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/78074

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1157923

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.