Ironic jobs failing in tripleo CI due to extra specs error

[Gregory Haynes]

Seeng these log entries in nova-compute:

Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] Traceback (most recent call last):
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2158, in _build_resources
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] yield resources
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2037, in _build_and_run_instance
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] block_device_info=block_device_info)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 577, in spawn
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] self._add_driver_fields(node, instance, image_meta, flavor)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 270, in _add_driver_fields
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] preserve_ephemeral)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/patcher.py", line 140, in get_deploy_patch
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] for key, value in self._get_kernel_ramdisk_dict(flavor).items():
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/patcher.py", line 109, in _get_kernel_ramdisk_dict
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] extra_specs = flavor['extra_specs']
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 467, in __getitem__
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] return getattr(self, name)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 70, in getter
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] self.obj_load_attr(name)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/flavor.py", line 86, in obj_load_attr
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] action='obj_load_attr', reason='unable to load %s' % attrname)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] ObjectActionError: Object action obj_load_attr failed because: unable to load extra_specs
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34]
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.566 4294 WARNING ironic.nova.virt.ironic.driver [req-dc62cada-b244-4364-8103-11d7c3b80f90 None] Destroy called on non-existing instance 102e1f91-83fe-426c-999f-e8d56af1dc34.

[Derek Higgins (derekh)]

A revert of Ironic https://review.openstack.org/#/c/116772/
Change-Id: Idfe0730daa22c6550a1a15b45f481635d2eb5195
fixes the problem

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/119357

[Dan Prince (dan-prince)]

Okay. So a couple of related Ironic driver breakages which are related here:

In git 17b08b421625ee6de332ecb031d866faa77ad850 we added a fix to arch.canonicalize so it automatically converts 'amd64' to 'X86_64'.

Also, I've pushed a fix so that we don't canonicalize the arch for extra_specs. See here:

https://review.openstack.org/#/c/120099/

Finally the Ironic compute driver (in the Ironic tree) is being deprecated such that it will simply subclass the Nova driver. See here: https://review.openstack.org/#/c/119892/

Once all of these changes land I think we should be good to go again.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Derek Higgins (<email address hidden>) on branch: master
Review: https://review.openstack.org/119357

[Ben Nemec (bnemec)]

The two changes Dan linked above have _not_ fixed this for me. I set my Nova reporefs to pull down the pending Nova change and the Ironic change already merged, but my local devtest run is still failing with the extra_specs exception. I can see both changes in my venv code so I don't think I messed up the reporefs (this time...).

[Dmitry Tantsur (divius)]

Should be fixed in nova now