tempest baremetal client is creating node with wrong property keys

[Adam Gandelman]

A new test has been added to tempest to stress the os-baremetal-nodes API extension. The test periodically fails in the gate with traceback in n-api log:

[req-01dcd35b-55f4-4688-ba18-7fe0c6defd52 BaremetalNodesAdminTestJSON-1864409967 BaremetalNodesAdminTestJSON-1481542636] Caught error: 'cpus'
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack Traceback (most recent call last):
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return req.get_response(self.application)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack application, catch_exc_info=False)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return resp(environ, start_response)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 977, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 902, in _call_app
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 749, in __call__
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack content_type, body, accept)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 814, in _process_stack
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 904, in dispatch
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack return method(req=request, **action_args)
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/contrib/baremetal_nodes.py", line 123, in index
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack 'cpus': inode.properties['cpus'],
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack KeyError: 'cpus'
2015-02-19 01:37:41.910 2521 TRACE nova.api.openstack

This hits only periodically and only when another tempest baremetal test is running in parallel to the new test. The other tests (tempest.api.baremetal.*) create some nodes in Ironic with node properties that are not the standard resource properties the nova->ironic proxy expects (from nova/api/openstack/compute/contrib/baremetal_nodes.py:201):

          for inode in ironic_nodes:
                node = {'id': inode.uuid,
                        'interfaces': [],
                        'host': 'IRONIC MANAGED',
                        'task_state': inode.provision_state,
                        'cpus': inode.properties['cpus'],
                        'memory_mb': inode.properties['memory_mb'],
                        'disk_gb': inode.properties['local_gb']}
                nodes.append(node)

The other tempest test is creating nodes with a different set of properties:

 {u'cpu_arch': u'x86', u'cpu_num': 8, u'memory': 4096, u'storage': 1024}

Ironic node properties are meant to be an arbitrary json dict, so the nova code needs to account for this in one way or another.

[Matt Riedemann (mriedem)]

We should probably just do something like this:

'cpus': inode.properties.get('cpus') or inode.properties.get('cpu_num'),

[Matt Riedemann (mriedem)]

These are the baremetal tests added to tempest on 2/16 when the failure started:

https://github.com/openstack/tempest/commit/ac0879accd2a451c54ed935297a6f8e0a8aaf25a

And looking at the tempest api schema checks and the nova code, they match what's in the ironic API:

http://docs.openstack.org/developer/ironic/webapi/v1.html#nodes

So I'm not convinced now that this is a nova bug, maybe ironic is passing back different keys? Or tempest is somehow setting up bad data?

[Matt Riedemann (mriedem)]

The baremetal client in tempest is creating the node with the wrong data:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/services/baremetal/v1/json/baremetal_client.py#n134

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/157256

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/157256
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=3ea1eb81d557a9a86553960f2bdb39ccc99787ea
Submitter: Jenkins
Branch: master

commit 3ea1eb81d557a9a86553960f2bdb39ccc99787ea
Author: Adam Gandelman <email address hidden>
Date: Wed Feb 18 19:13:25 2015 -0800

    Fix baremetal node property keys

    The baremetal_client is creating test nodes with property keys that
    don't match what the nova os-baremetal-nodes extension is pulling out of
    the node properties from the ironic client. This results in KeyErrors
    when this test runs in parallel to other baremetal tests that were added
    to Tempest on 2/16 with commit ac0879accd2a451c54ed935297a6f8e0a8aaf25a.

    Closes-Bug: #1423427

    Co-authored-by: Adam Gandelman <email address hidden>

    Change-Id: I3593f58de5088d2442fddd52c6a11d055bb4f6a0