Ironic jobs failing in tripleo CI due to extra specs error

Bug #1365750 reported by Gregory Haynes on 2014-09-04
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ironic
Invalid
Undecided
Unassigned
OpenStack Compute (nova)
Undecided
Unassigned
tripleo
Critical
Unassigned

Bug Description

Seeng these log entries in nova-compute:

Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] Traceback (most recent call last):
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2158, in _build_resources
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] yield resources
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2037, in _build_and_run_instance
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] block_device_info=block_device_info)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 577, in spawn
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] self._add_driver_fields(node, instance, image_meta, flavor)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 270, in _add_driver_fields
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] preserve_ephemeral)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/patcher.py", line 140, in get_deploy_patch
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] for key, value in self._get_kernel_ramdisk_dict(flavor).items():
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/patcher.py", line 109, in _get_kernel_ramdisk_dict
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] extra_specs = flavor['extra_specs']
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 467, in __getitem__
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] return getattr(self, name)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 70, in getter
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] self.obj_load_attr(name)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/flavor.py", line 86, in obj_load_attr
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] action='obj_load_attr', reason='unable to load %s' % attrname)
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34] ObjectActionError: Object action obj_load_attr failed because: unable to load extra_specs
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.284 4294 TRACE nova.compute.manager [instance: 102e1f91-83fe-426c-999f-e8d56af1dc34]
Sep 4 19:25:34 ubuntu nova-compute: 2014-09-04 19:25:34.566 4294 WARNING ironic.nova.virt.ironic.driver [req-dc62cada-b244-4364-8103-11d7c3b80f90 None] Destroy called on non-existing instance 102e1f91-83fe-426c-999f-e8d56af1dc34.

Changed in ironic:
status: New → Confirmed
status: Confirmed → New
Changed in tripleo:
status: New → Confirmed
importance: Undecided → Critical
Derek Higgins (derekh) wrote :

A revert of Ironic https://review.openstack.org/#/c/116772/
Change-Id: Idfe0730daa22c6550a1a15b45f481635d2eb5195
fixes the problem

Dan Prince (dan-prince) wrote :

Okay. So a couple of related Ironic driver breakages which are related here:

In git 17b08b421625ee6de332ecb031d866faa77ad850 we added a fix to arch.canonicalize so it automatically converts 'amd64' to 'X86_64'.

Also, I've pushed a fix so that we don't canonicalize the arch for extra_specs. See here:

https://review.openstack.org/#/c/120099/

Finally the Ironic compute driver (in the Ironic tree) is being deprecated such that it will simply subclass the Nova driver. See here: https://review.openstack.org/#/c/119892/

Once all of these changes land I think we should be good to go again.

Change abandoned by Derek Higgins (<email address hidden>) on branch: master
Review: https://review.openstack.org/119357

Ben Nemec (bnemec) wrote :

The two changes Dan linked above have _not_ fixed this for me. I set my Nova reporefs to pull down the pending Nova change and the Ironic change already merged, but my local devtest run is still failing with the extra_specs exception. I can see both changes in my venv code so I don't think I messed up the reporefs (this time...).

Dmitry Tantsur (divius) wrote :

Should be fixed in nova now

tags: added: ironic
Changed in ironic:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers