Cleaning fails when nodes have traits

[Mark Goddard]

If cleaning is performed on a node that has traits assigned, then cleaning will fail, and the node will end up in CLEAN_FAILED.

Affects master & stable/queens.

Error [1] seen during cleaning while adding traits to the nodes in devstack.

Looks to be an issue with attempting JSON serialisation of trait RPC objects.

Traceback:

Feb 16 16:36:43.889351 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: /usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py:192: UserWarning: Cannot convert Trait(created_at=2018-02-16T16:34:14Z,node_id='1',trait='CUSTOM_GOLD',updated_at=None) to primitive, will raise ValueError instead of warning in version 3.0
Feb 16 16:36:43.889664 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: "instead of warning in version 3.0" % (value,))
Feb 16 16:36:43.894193 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception: Node failed to start the first cleaning step. Exception: Circular reference detected for node 6a420cf7-6b98-4fc2-a70b-f78deb64e4a4: ValueError: Circular reference detected
Feb 16 16:36:43.894467 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
Feb 16 16:36:43.894717 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 328, in heartbeat
Feb 16 16:36:43.895005 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor self.refresh_clean_steps(task)
Feb 16 16:36:43.895255 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.895497 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.895798 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_base_vendor.py", line 405, in refresh_clean_steps
Feb 16 16:36:43.896022 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor agent_result = self._client.get_clean_steps(node, task.ports).get(
Feb 16 16:36:43.896256 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.896464 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.896672 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 179, in get_clean_steps
Feb 16 16:36:43.896886 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor wait=True)
Feb 16 16:36:43.897122 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
Feb 16 16:36:43.897330 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
Feb 16 16:36:43.897550 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 58, in _command
Feb 16 16:36:43.897766 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor body = self._get_command_body(method, params)
Feb 16 16:36:43.897985 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/opt/stack/new/ironic/ironic/drivers/modules/agent_client.py", line 52, in _get_command_body
Feb 16 16:36:43.898197 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor 'params': params,
Feb 16 16:36:43.898406 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 213, in dumps
Feb 16 16:36:43.898619 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return json.dumps(obj, default=default, **kwargs)
Feb 16 16:36:43.898829 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
Feb 16 16:36:43.899033 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor sort_keys=sort_keys, **kw).encode(obj)
Feb 16 16:36:43.899244 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
Feb 16 16:36:43.899457 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor chunks = self.iterencode(o, _one_shot=True)
Feb 16 16:36:43.899675 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
Feb 16 16:36:43.899890 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor return _iterencode(o, 0)
Feb 16 16:36:43.900099 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor ValueError: Circular reference detected
Feb 16 16:36:43.900309 ubuntu-xenial-rax-iad-0002580271 ironic-conductor[20042]: ERROR ironic.drivers.modules.agent_base_vendor

[1] http://logs.openstack.org/70/545370/1/check/ironic-tempest-dsvm-ipa-partition-uefi-pxe_ipmitool-tinyipa/d5486a9/logs/screen-ir-cond.txt.gz#_Feb_16_16_36_43_889351

[John L. Villalovos (happycamp)]

Maybe there needs to be a __repr__() method defined

[Dmitry Tantsur (divius)]

It feels like we're injecting a raw Trait object somewhere in the JSON.

[Mark Goddard (mgoddard)]

Yes, that's correct. The problem is that objects.IronicObject.as_dict() doesn't handle nested objects properly - it needs to call .as_dict() on these.

Working on some changes, which *mostly* seem to work except for one unit test.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545830

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/546830

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546830
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=be94e64b912bfdb70c578918815a28ed8611662a
Submitter: Zuul
Branch: stable/queens

commit be94e64b912bfdb70c578918815a28ed8611662a
Author: Mark Goddard <email address hidden>
Date: Mon Feb 19 13:29:45 2018 +0000

    Support nested objects and object lists in as_dict

    The value returned by ironic.objects.IronicObject.as_dict() should be a
    plain object, in order for it to be serialised to JSON. Currently,
    nested object fields and object list fields are not converted to dict
    format. This caused problems during cleaning, when the node object's
    as_dict representation is JSON encoded and sent to IPA.

    This change adds support for calling as_dict() on nested objects and
    list objects, to ensure these are also returned in dict form.

    We also change the method used in as_dict() for checking whether an
    object has an attribute. The hasattr() function used previously has
    problems when used with properties in python 2 [1], in that any
    exceptions raised in the property getter result in hasattr() returning
    False. Instead we use obj_attr_is_set() to determine whether the object
    has a particular attribute.

    [1] https://hynek.me/articles/hasattr/

    Change-Id: Ib2166040508827db28d6f6e2d9a3e655c16f2993
    Closes-Bug: #1750027
    (cherry picked from commit c66679f14b0b5f34462a896c3bff5a9f483a9a83)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 10.1.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545370
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=c8ae2457c7fd86975d77d6bc67b50b155f602944
Submitter: Zuul
Branch: master

commit c8ae2457c7fd86975d77d6bc67b50b155f602944
Author: Mark Goddard <email address hidden>
Date: Fri Feb 16 15:59:23 2018 +0000

    Register traits on nodes in devstack

    This change adds support for assiging traits to nodes in the ironic
    devstack plugin. The set of traits added is configured by
    IRONIC_DEFAULT_TRAITS, which defaults to a single trait - CUSTOM_GOLD

    Using nodes with traits during testing ensures that we are exercising
    node traits code paths in the Ironic CI.

    Node traits are not used for upgrade testing, as the queens devstack
    plugin does not currently register traits. Once this change is
    backported to queens, we can enable traits during upgrade testing.

    Change-Id: I18b9dbf8a5b16e36e0cc14f290b12ffdd634364a
    Related-Bug: #1722194
    Related-Bug: #1750027

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/551938

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/551938
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=6015890adee55582330256bc10e933cba0b84cc9
Submitter: Zuul
Branch: stable/queens

commit 6015890adee55582330256bc10e933cba0b84cc9
Author: Mark Goddard <email address hidden>
Date: Fri Feb 16 15:59:23 2018 +0000

    Register traits on nodes in devstack

    This change adds support for assiging traits to nodes in the ironic
    devstack plugin. The set of traits added is configured by
    IRONIC_DEFAULT_TRAITS, which defaults to a single trait - CUSTOM_GOLD

    Using nodes with traits during testing ensures that we are exercising
    node traits code paths in the Ironic CI.

    Node traits are not used for upgrade testing, as the queens devstack
    plugin does not currently register traits. Once this change is
    backported to queens, we can enable traits during upgrade testing.

    Change-Id: I18b9dbf8a5b16e36e0cc14f290b12ffdd634364a
    Related-Bug: #1722194
    Related-Bug: #1750027
    (cherry picked from commit c8ae2457c7fd86975d77d6bc67b50b155f602944)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/552077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/552077
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=cef19cbf0a4d7b179de5d4448e247e9db1a79831
Submitter: Zuul
Branch: master

commit cef19cbf0a4d7b179de5d4448e247e9db1a79831
Author: Mark Goddard <email address hidden>
Date: Mon Mar 12 16:18:51 2018 +0000

    Use node traits during upgrade

    Now that the queens branch has support for node traits in the devstack
    plugin [1], we can apply node traits when testing upgrades from queens
    or later.

    [1] https://review.openstack.org/551938

    Change-Id: I99b9487d849ecd5241657d6ca91c2e9f9f7dfcf8
    Related-Bug: #1722194
    Related-Bug: #1750027

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 11.0.0 release.