| 2018-02-19 20:15:37 |
Mark Goddard |
description |
Quick bug, need to head out.
Error [1] seen during cleaning while adding traits to the nodes in devstack.
Looks to be an issue with attempting JSON serialisation of trait RPC objects.
Traceback:
Feb 16 16:36:43.889351 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: /usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py:192: UserWarning: Cannot convert Trait(created_at=2018-02-16T16:34:14Z,node_id='1',trait='CUSTOM_GOLD',updated_at=None) to primitive, will raise ValueError instead of warning in version 3.0
Feb 16 16:36:43.889664 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: "instead of warning in version 3.0" % (value,))
Feb 16 16:36:43.894193 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception: Node failed to start the first cleaning step. Exception: Circular reference detected for node 6a420cf7-6b98-4fc2-a70b-f78deb64e4a4: ValueError: Circular reference detected
Feb 16 16:36:43.894467 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
Feb 16 16:36:43.894717 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 328, in heartbeat
Feb 16 16:36:43.895005 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor self.refresh_clean_steps(task)
Feb 16 16:36:43.895255 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.895497 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.895798 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 405, in refresh_clean_steps
Feb 16 16:36:43.896022 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor agent_result = self._client.get_clean_steps(node, task.ports).get(
Feb 16 16:36:43.896256 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.896464 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.896672 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 179, in get_clean_steps
Feb 16 16:36:43.896886 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor wait=True)
Feb 16 16:36:43.897122 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.897330 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.897550 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 58, in _command
Feb 16 16:36:43.897766 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor body = self._get_command_body(method, params)
Feb 16 16:36:43.897985 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 52, in _get_command_body
Feb 16 16:36:43.898197 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor 'params': params,
Feb 16 16:36:43.898406 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 213, in dumps
Feb 16 16:36:43.898619 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return json.dumps(obj, default=default, **kwargs)
Feb 16 16:36:43.898829 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
Feb 16 16:36:43.899033 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor sort_keys=sort_keys, **kw).encode(obj)
Feb 16 16:36:43.899244 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
Feb 16 16:36:43.899457 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor chunks = self.iterencode(o, _one_shot=True)
Feb 16 16:36:43.899675 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
Feb 16 16:36:43.899890 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return _iterencode(o, 0)
Feb 16 16:36:43.900099 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor ValueError: Circular reference detected
Feb 16 16:36:43.900309 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor
[1] http://logs.openstack.org/70/545370/1/check/ironic-tempest-dsvm-ipa-partition-uefi-pxe_ipmitool-tinyipa/d5486a9/logs/screen-ir-cond.txt.gz#_Feb_16_16_36_43_889351 |
If cleaning is performed on a node that has traits assigned, then cleaning will fail, and the node will end up in CLEAN_FAILED.
Affects master & stable/queens.
Error [1] seen during cleaning while adding traits to the nodes in devstack.
Looks to be an issue with attempting JSON serialisation of trait RPC objects.
Traceback:
Feb 16 16:36:43.889351 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: /usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py:192: UserWarning: Cannot convert Trait(created_at=2018-02-16T16:34:14Z,node_id='1',trait='CUSTOM_GOLD',updated_at=None) to primitive, will raise ValueError instead of warning in version 3.0
Feb 16 16:36:43.889664 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: "instead of warning in version 3.0" % (value,))
Feb 16 16:36:43.894193 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception: Node failed to start the first cleaning step. Exception: Circular reference detected for node 6a420cf7-6b98-4fc2-a70b-f78deb64e4a4: ValueError: Circular reference detected
Feb 16 16:36:43.894467 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
Feb 16 16:36:43.894717 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 328, in heartbeat
Feb 16 16:36:43.895005 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor self.refresh_clean_steps(task)
Feb 16 16:36:43.895255 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.895497 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.895798 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 405, in refresh_clean_steps
Feb 16 16:36:43.896022 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor agent_result = self._client.get_clean_steps(node, task.ports).get(
Feb 16 16:36:43.896256 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.896464 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.896672 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 179, in get_clean_steps
Feb 16 16:36:43.896886 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor wait=True)
Feb 16 16:36:43.897122 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.897330 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.897550 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 58, in _command
Feb 16 16:36:43.897766 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor body = self._get_command_body(method, params)
Feb 16 16:36:43.897985 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 52, in _get_command_body
Feb 16 16:36:43.898197 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor 'params': params,
Feb 16 16:36:43.898406 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 213, in dumps
Feb 16 16:36:43.898619 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return json.dumps(obj, default=default, **kwargs)
Feb 16 16:36:43.898829 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
Feb 16 16:36:43.899033 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor sort_keys=sort_keys, **kw).encode(obj)
Feb 16 16:36:43.899244 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
Feb 16 16:36:43.899457 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor chunks = self.iterencode(o, _one_shot=True)
Feb 16 16:36:43.899675 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
Feb 16 16:36:43.899890 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return _iterencode(o, 0)
Feb 16 16:36:43.900099 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor ValueError: Circular reference detected
Feb 16 16:36:43.900309 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor
[1] http://logs.openstack.org/70/545370/1/check/ironic-tempest-dsvm-ipa-partition-uefi-pxe_ipmitool-tinyipa/d5486a9/logs/screen-ir-cond.txt.gz#_Feb_16_16_36_43_889351 |
|
