Nova boot fails: AttributeError: is_public

[Cory Stone]

[req-d4c97a98-2b0e-419e-83d6-0e88332a699a account1 account1] [instance: 036a26b1-7fe2-4d56-b7a2-4781e8cad696] Error: is_public
Traceback (most recent call last):
  File "/opt/nova/nova/compute/manager.py", line 1254, in _build_instance
    set_access_ip=set_access_ip)
  File "/opt/nova/nova/compute/manager.py", line 394, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/nova/nova/compute/manager.py", line 1655, in _spawn
    LOG.exception(_('Instance failed to spawn'), instance=instance)
  File "/opt/nova/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/nova/nova/compute/manager.py", line 1652, in _spawn
    block_device_info)
  File "/opt/nova/nova/virt/libvirt/driver.py", line2230, in spawn
    admin_pass=admin_password)
  File "/opt/nova/nova/virt/libvirt/driver.py", line2538, in _create_image
    imagehandler_args=imagehandler_args)
  File "/opt/nova/nova/virt/libvirt/imagebackend.py", line 184, in cache
    *args, **kwargs)
  File "/opt/nova/nova/virt/libvirt/imagebackend.py", line 310, in create_image
    prepare_template(target=base, max_size=size, *args, **kwargs)
  File "/opt/nova/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)
  File "/opt/nova/nova/virt/libvirt/imagebackend.py", line 174, in fetch_func_sync
    fetch_func(target=target, *args, **kwargs)
  File "/opt/nova/nova/virt/libvirt/utils.py", line 654, in fetch_image
    max_size=max_size, imagehandler_args=imagehandler_args)
  File "/opt/nova/nova/virt/images.py", line 108, in fetch_to_raw
    imagehandler_args=imagehandler_args)
  File "/opt/nova/nova/virt/images.py", line 98, in fetch
    fetched_to_local = handler.is_local()
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/nova/nova/openstack/common/fileutils.py", line 98, in remove_path_on_error
    remove(path)
  File "/opt/nova/nova/virt/images.py", line 71, in _remove_image_on_exec
    image_path):
  File "/opt/nova/nova/virt/imagehandler/__init__.py", line 154, in handle_image
    img_locs = image_service.get_locations(context, image_id)
  File "/opt/nova/nova/image/glance.py", line 307, in get_locations
    if not self._is_image_available(context, image_meta):
  File "/opt/nova/nova/image/glance.py", line 446, in _is_image_available
    if image.is_public or context.is_admin:
  File "/usr/local/lib/python2.7/dist-packages/warlock/model.py", line 72, in __getattr__
    raise AttributeError(key)
AttributeError: is_public

[Cory Stone (corystone)]

Here's the entire nova.conf:

[DEFAULT]
debug=True
verbose=True
nodaemon=True
compute_driver=libvirt.LibvirtDriver
libvirt_type=qemu
network_manager=nova.network.manager.FlatDHCPManager
dhcpbridge_flagfile=/etc/nova/nova.conf
flat_network_bridge=virbr0
glance_api_servers=glance:9292
iscsi_ip_prefix=10.
state_path=/etc/nova/state
lock_path=/etc/nova/lock
logdir=/var/log/nova/
volume_api_class=nova.volume.cinder.API
cinder_endpoint_template=http://lunr:8776/v1/%(project_id)s
sql_connection=mysql://root:@localhost/nova

[Matt Riedemann (mriedem)]

In IRC Cory said that if he reverts https://review.openstack.org/#/c/33409/ then it works, so there is probably some regression there, but we need some more configuration details since Cory isn't running with the default devstack.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/79732

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/79733

[Cory Stone (corystone)]

Glance is from the ubuntu cloud archive: 1:2013.2.2-0ubuntu1~cloud0.

glance-api.conf

[DEFAULT]
verbose = True
debug = False
default_store = file
bind_host = 0.0.0.0
bind_port = 9292
log_file = /var/log/glance/api.log
backlog = 4096
sql_connection = sqlite:////var/lib/glance/glance.sqlite
sql_idle_timeout = 3600
workers = 1
registry_host = 0.0.0.0
registry_port = 9191
registry_client_protocol = http
notifier_strategy = noop
rabbit_host = localhost
rabbit_port = 5672
rabbit_use_ssl = false
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_notification_exchange = glance
rabbit_notification_topic = notifications
rabbit_durable_queues = False
qpid_notification_exchange = glance
qpid_notification_topic = notifications
qpid_hostname = localhost
qpid_port = 5672
qpid_username =
qpid_password =
qpid_sasl_mechanisms =
qpid_reconnect_timeout = 0
qpid_reconnect_limit = 0
qpid_reconnect_interval_min = 0
qpid_reconnect_interval_max = 0
qpid_reconnect_interval = 0
qpid_heartbeat = 5
qpid_protocol = tcp
qpid_tcp_nodelay = True
filesystem_store_datadir = /var/lib/glance/images/
swift_store_auth_version = 2
swift_store_auth_address = 127.0.0.1:5000/v2.0/
swift_store_user = jdoe:jdoe
swift_store_key = a86850deb2742ec3cb41518e26aa2d89
swift_store_container = glance
swift_store_create_container_on_put = False
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False

[Jay Pipes (jaypipes)]

Note that I don't think this actually has something to do with the patch series that is proposed for reversion. This bug has existed in Nova for a while.

On line 446 of nova.images.glance:

 File "/opt/nova/nova/image/glance.py", line 446, in _is_image_available
    if image.is_public or context.is_admin:

The problem is that image.is_public is only relevant for Glance v1 API. In Glance v2 API, the check is image.visibility == 'public', IIRC, so it is nova.image.glance module that needs to be changed to account for the non-backwards-compatible way that the v1 and v2 glanceclient result proxies work.

[Cory Stone (corystone)]

My initial basic printf debugging from inside _is_image_available() before the stack trace:

LOG.warn(_("image class name: %s") % image.__class__.__name__)
LOG.warn(_("dir(image): %s") % str(dir(image)))

Results in:

 WARNING nova.image.glance [req-66cf735b-0f84-4e00-863d-ca8c8a5fc04c account1 account1] image class name: image
 WARNING nova.image.glance [req-66cf735b-0f84-4e00-863d-ca8c8a5fc04c account1 account1] dir(image): ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__original__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'changes', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'patch', 'pop', 'popitem', 'schema', 'setdefault', 'update', 'validate', 'values', 'viewitems', 'viewkeys', 'viewvalues']

A dict?

[Matt Riedemann (mriedem)]

FYI, none of the patches to make nova support glance v2 landed:

https://blueprints.launchpad.net/nova/+spec/use-glance-v2-api

So maybe an issue with the multiple image locations support assuming glance v2 is being used?

[Matt Riedemann (mriedem)]

Jay Pipes found it:

https://github.com/openstack/nova/blob/master/nova/image/glance.py#L301

There is a hard-coded '2' for the glance v2 API in that code which isn't going to work given the above blueprint never landed in nova.

[Dmitry Borodaenko (angdraug)]

That '2' is hardcoded there for a reason: get_location needs direct_url that is only available in Glance v2 API:
https://review.openstack.org/13017

I think Jay's comment above is spot on: _is_image_available shouldn't assume v1 API.

[Matt Riedemann (mriedem)]

Patch is up: https://review.openstack.org/#/c/79780/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79780
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8babd6a99014ccaf51d955769eaec085e037cc76
Submitter: Jenkins
Branch: master

commit 8babd6a99014ccaf51d955769eaec085e037cc76
Author: Jay Pipes <email address hidden>
Date: Tue Mar 11 17:29:37 2014 -0400

    Ensure is_image_available handles V2 Glance API

    The V2 Glance API uses warlock to validate returned models
    from calls such as GET /images/{id}. The get_locations() call in
    the Nova Glance image driver was using the V2 Glance API call
    to GET /images/{id} (which is correct, since image locations only
    exist in the V2 API), but the GlanceImageService._is_image_available()
    method was checking the is_public attribute of the supplied image model.

    This caused warlock to throw an exception since is_public is not
    an attribute on the V2 API's returned image model in glanceclient.
    This change adds some checks that can handle both the V2 and V1 returned
    image models from glanceclient and a bunch of unit tests to thoroughly
    exercise the is_image_available method.

    Change-Id: I35b5af8331c5972bd89e1a674091fa2e3bde28a7
    Closes-bug: #1291014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79732
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cef7dd6a8fcd1e4ab148dd228129e267b1e82232
Submitter: Jenkins
Branch: master

commit cef7dd6a8fcd1e4ab148dd228129e267b1e82232
Author: Dan Smith <email address hidden>
Date: Tue Mar 11 12:09:23 2014 -0700

    Revert "enable cloning for rbd-backed ephemeral disks"

    This reverts commit c25c60f6a9ab1ccf12f72f76d400e7c9c0d090b3.

    The series of patches involved with adding this feature introduced
    an unexpected dependency on glance's v2 API, which we do not
    currently support. Triggering a user-facing bug quickly, and leaving
    some uncertainty about what else is likely to come in the future,
    a revert of this code was decided given the short time to -rc1.

    Change-Id: Iee835652f56beed1b8ac74d782e34a5238102840
    Related-bug: 1291014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79733
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a7e018e8c76d6805dc0e857f24c64e3223224ae8
Submitter: Jenkins
Branch: master

commit a7e018e8c76d6805dc0e857f24c64e3223224ae8
Author: Dan Smith <email address hidden>
Date: Tue Mar 11 12:10:13 2014 -0700

    Revert "Move libvirt RBD utilities to a new file"

    This reverts commit cb8968400f8c8370cab309de0f63e82565bfb25b.

    The series of patches involved with adding this feature introduced
    an unexpected dependency on glance's v2 API, which we do not
    currently support. Triggering a user-facing bug quickly, and leaving
    some uncertainty about what else is likely to come in the future,
    a revert of this code was decided given the short time to -rc1.

    Change-Id: I5e5d897f7294a7d4bda76ed392f503d0552f45e9
    Related-bug: 1291014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79734
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0ba9cca788309bd8763a0d277a907769604cd9f4
Submitter: Jenkins
Branch: master

commit 0ba9cca788309bd8763a0d277a907769604cd9f4
Author: Dan Smith <email address hidden>
Date: Tue Mar 11 12:11:05 2014 -0700

    Revert "Adding image multiple location support"

    This reverts commit a55bbbfa1957d02b3006a51afcf4984a1f8ff886.

    The series of patches involved with adding this feature introduced
    an unexpected dependency on glance's v2 API, which we do not
    currently support. Triggering a user-facing bug quickly, and leaving
    some uncertainty about what else is likely to come in the future,
    a revert of this code was decided given the short time to -rc1.

    Closes-bug: 1291014
    Change-Id: I2ed6a861e583b9513b0984ff9801d4b9f7536798