Doc fix needed around local image store (was Image not found on nova-compute only node)

[Kevin Jackson]

In a multi-node setup where I have 1 node running all services (compute,api,network,objectstore,scheduler) and another running compute - when an instance tries to spawn on the compute only node it fails to spawn with the following trace:

2011-03-17 10:28:10,358 DEBUG nova.utils [-] Attempting to grab iptables for method apply... from (pid=1458) inner /usr/lib/pymodules/python2.6/nova/utils.py:535
2011-03-17 10:28:10,370 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,389 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,410 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,427 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,455 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000042/ from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,467 INFO nova.virt.libvirt_conn [-] instance instance-00000042: Creating image
2011-03-17 10:28:10,569 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/5ee40cc4 /var/lib/nova/instances/instance-00000042/kernel from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,613 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/310e4e6b /var/lib/nova/instances/instance-00000042/ramdisk from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 10:28:10,670 ERROR nova.compute.manager [YM1X-1RU8X8VLFWWWFAL kevinj myproject] instance 66: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 218, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.compute.manager): TRACE: return f(*args, **kw)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 424, in spawn
(nova.compute.manager): TRACE: self._create_image(instance, xml)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 638, in _create_image
(nova.compute.manager): TRACE: size=size)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 559, in _cache_image
(nova.compute.manager): TRACE: fn(target=base, *args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 572, in _fetch_image
(nova.compute.manager): TRACE: images.fetch(image_id, target, user, project)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/images.py", line 51, in fetch
(nova.compute.manager): TRACE: metadata = image_service.get(elevated, image_id, image_file)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/image/local.py", line 95, in get
(nova.compute.manager): TRACE: raise exception.NotFound
(nova.compute.manager): TRACE: NotFound: None
(nova.compute.manager): TRACE:

Running an instance on the first node works fine.

[pjordan (j0rd4n)]

I can confirm this bug. Exact same stack trace.

Nova bzr815.

[Vish Ishaya (vishvananda)]

Local Image service is exactly that, a local image service. If you want multiple machines to access the images locally, you will need to share the local image service folder to all of your compute hosts via nfs or something like that. The recommended setup for a multiple machine service would be to install and run glance and use:
--image_service=nova.image.glance.GlanceImageService

Vish

On Mar 17, 2011, at 5:12 AM, pjordan wrote:

> I can confirm this bug. Exact same stack trace.
>
> Nova bzr815.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/736754
>
> Title:
> Image not found when instance tries to spawn on nova-compute only node
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> In a multi-node setup where I have 1 node running all services
> (compute,api,network,objectstore,scheduler) and another running
> compute - when an instance tries to spawn on the compute only node it
> fails to spawn with the following trace:
>
> 2011-03-17 10:28:10,358 DEBUG nova.utils [-] Attempting to grab iptables for method apply... from (pid=1458) inner /usr/lib/pymodules/python2.6/nova/utils.py:535
> 2011-03-17 10:28:10,370 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,389 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,410 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,427 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,455 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000042/ from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,467 INFO nova.virt.libvirt_conn [-] instance instance-00000042: Creating image
> 2011-03-17 10:28:10,569 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/5ee40cc4 /var/lib/nova/instances/instance-00000042/kernel from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,613 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/310e4e6b /var/lib/nova/instances/instance-00000042/ramdisk from (pid=1458) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
> 2011-03-17 10:28:10,670 ERROR nova.compute.manager [YM1X-1RU8X8VLFWWWFAL kevinj myproject] instance 66: Failed to spawn
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 218, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, i...

[Kevin Jackson (kevin-linuxservices)]

Thanks for clarifying (it makes sense) - I think the issue then is of documentation or some other assumption because the default isn't what the documentation says.

http://docs.openstack.org/openstack-compute/admin/content/ch05s08.html

This says the following, where the default should be S3 as what was working before upgrading to this release.

The service to use for retrieving and searching for images. Images must be registered using euca2ools. Options:

    *

      nova.image.s3.S3ImageService

      S3 backend for the imaging service; This is the default setting if no image manager is defined here.
    *

      nova.image.local.LocalImageService

      Image service storing images to local disk. It assumes that image_ids are integers.
    *

      nova.image.glance.GlanceImageService

      Glance back end for storing and retrieving images; See http://glance.openstack.org for more info.

[Josh Kearney (jk0)]

Pulled the latest openstack-manuals trunk and this seems to have been fixed.

[Anne Gentle (annegentle)]

Yep, I fixed it in source, but don't have it published to the site yet so yeah, Fix Committed is right. Sorry to have you chase it, but thanks for the assist!