utf-8 codec can't decode lldp data breaking node inspection

[Riccardo Pittau]

in case the lldp raw data collected by the inspection process includes non utf-8 information, the parser fails breaking the inspection process
For example:
023-11-22 18:27:04.664 1 ERROR ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state processing] Processing the error event because of an exception <class 'UnicodeDecodeError'>: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte raised by ironic_inspector.process._pro
cess_node: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byteESC[00m
2023-11-22 18:27:04.664 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state processing] Successfully released lock release_lock /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:125ESC[00m
2023-11-22 18:27:04.670 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state processing] Executing fsm(processing).process_event(error) fsm_event /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:200ESC[00m
2023-11-22 18:27:04.670 1 INFO ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state processing] Updating node state: processing --> errorESC[00m
2023-11-22 18:27:04.671 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state processing] Committing fields: {'state': 'error'} _commit /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:142ESC[00m
2023-11-22 18:27:04.673 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state error] Committing fields: {'finished_at': datetime.datetime(2023, 11, 22, 18, 27, 4, 665103), 'error': "'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte"} _commit /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:142ESC[00m
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process [-] Unexpected exception during processing: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process Traceback (most recent call last):
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/process.py", line 237, in process
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process result = _process_node(node_info, node, introspection_data)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py", line 582, in inner
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return func(node_info, *args, **kwargs)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py", line 560, in inner
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process node_info.finished(istate.Events.error, error=str(exc))
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process self.force_reraise()
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process raise self.value
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py", line 544, in inner
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process ret = func(node_info, *args, **kwargs)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/process.py", line 271, in _process_node
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process _run_post_hooks(node_info, introspection_data)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/process.py", line 263, in _run_post_hooks
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process hook_ext.obj.before_update(introspection_data, node_info)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/plugins/lldp_basic.py", line 81, in before_update
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process nv = self._parse_lldp_tlvs(tlvs, node_info)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/plugins/lldp_basic.py", line 56, in _parse_lldp_tlvs
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process if parser.parse_tlv(tlv_type, data):
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/ironic_inspector/common/lldp_parsers.py", line 161, in parse_tlv
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process struct = tlv_parser.parse(data)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 288, in parse
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return self.parse_stream(io.BytesIO(data), **contextkw)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 300, in parse_stream
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return self._parsereport(stream, context, "(parsing)")
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 312, in _parsereport
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process obj = self._parse(stream, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 1981, in _parse
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process subobj = sc._parsereport(stream, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 312, in _parsereport
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process obj = self._parse(stream, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 2439, in _parse
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return self.subcon._parsereport(stream, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 312, in _parsereport
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process obj = self._parse(stream, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 698, in _parse
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return self._decode(obj, context, path)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process File "/usr/lib/python3.9/site-packages/construct/core.py", line 1495, in _decode
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process return obj.decode(self.encoding)
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte
2023-11-22 18:27:04.675 1 ERROR ironic_inspector.process ESC[00m
2023-11-22 18:27:04.685 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state error] Executing fsm(error).process_event(error) fsm_event /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:200ESC[00m
2023-11-22 18:27:04.686 1 DEBUG ironic_inspector.node_cache [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state error] Committing fields: {'finished_at': datetime.datetime(2023, 11, 22, 18, 27, 4, 681856), 'error': "Unexpected exception UnicodeDecodeError during processing: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte"} _commit /usr/lib/python3.9/site-packages/ironic_inspector/node_cache.py:142ESC[00m
2023-11-22 18:27:04.689 1 INFO ironic_inspector.process [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state error BMC 10.16.230.10] Ramdisk logs were stored in file f38df2c7-3a19-4f38-a3b7-385b2a971f53_20231122-182704.688370.tar.gzESC[00m
2023-11-22 18:27:04.689 1 ERROR ironic_inspector.utils [-] [node: f38df2c7-3a19-4f38-a3b7-385b2a971f53 state error BMC 10.16.230.10] Unexpected exception UnicodeDecodeError during processing: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byteESC[00m
2023-11-22 18:27:04.689 1 DEBUG ironic_inspector.main [None req-e587f885-ce93-45c0-86d2-0012f5bd4431 - - - - - -] Returning error to client: Unexpected exception UnicodeDecodeError during processing: 'utf-8' codec can't decode byte 0xf7 in position 13: invalid start byte error_response /usr/lib/python3.9/site-packages/ironic_inspector/main.py:139ESC[00m

This should be correctly handled in the inspector lldp plugins.
A possible solution is to add a try..except with a warning message at https://github.com/openshift/openstack-ironic-inspector/blob/release-4.14/ironic_inspector/plugins/lldp_basic.py#L56

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/903861

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/903861
Committed: https://opendev.org/openstack/ironic/commit/4d3101940a75120ffeb0f228d266534d4c74960c
Submitter: "Zuul (22348)"
Branch: master

commit 4d3101940a75120ffeb0f228d266534d4c74960c
Author: Riccardo Pittau <email address hidden>
Date: Mon Dec 18 10:19:22 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I48c18d1648d446d426515456a4006947b79a9ef3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: bugfix/23.1
Review: https://review.opendev.org/c/openstack/ironic/+/904860

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/904860
Committed: https://opendev.org/openstack/ironic/commit/6954653b483146fe56b80967265c6f8c7ce00453
Submitter: "Zuul (22348)"
Branch: bugfix/23.1

commit 6954653b483146fe56b80967265c6f8c7ce00453
Author: Riccardo Pittau <email address hidden>
Date: Mon Dec 18 10:19:22 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I48c18d1648d446d426515456a4006947b79a9ef3
    (cherry picked from commit 4d3101940a75120ffeb0f228d266534d4c74960c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic-inspector/+/903760
Committed: https://opendev.org/openstack/ironic-inspector/commit/db76af9c86bffb61f9b7a52a4080959fee1658fa
Submitter: "Zuul (22348)"
Branch: master

commit db76af9c86bffb61f9b7a52a4080959fee1658fa
Author: Riccardo Pittau <email address hidden>
Date: Fri Dec 15 15:55:00 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I939c80bb309b22e05ba7cc93686f12bbe0d71624

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: bugfix/11.8
Review: https://review.opendev.org/c/openstack/ironic-inspector/+/905340

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/ironic-inspector/+/905441

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/ironic-inspector/+/905442

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ironic-inspector/+/905443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic-inspector/+/905441
Committed: https://opendev.org/openstack/ironic-inspector/commit/38101c70e9ff3d3f7cf3902cfba3f88438414aba
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 38101c70e9ff3d3f7cf3902cfba3f88438414aba
Author: Riccardo Pittau <email address hidden>
Date: Fri Dec 15 15:55:00 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I939c80bb309b22e05ba7cc93686f12bbe0d71624
    (cherry picked from commit db76af9c86bffb61f9b7a52a4080959fee1658fa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic-inspector/+/905442
Committed: https://opendev.org/openstack/ironic-inspector/commit/b43ce63be3853c386e7bd8363b6a1f4e01650e06
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit b43ce63be3853c386e7bd8363b6a1f4e01650e06
Author: Riccardo Pittau <email address hidden>
Date: Fri Dec 15 15:55:00 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I939c80bb309b22e05ba7cc93686f12bbe0d71624
    (cherry picked from commit db76af9c86bffb61f9b7a52a4080959fee1658fa)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 24.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 12.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic-inspector/+/905443
Committed: https://opendev.org/openstack/ironic-inspector/commit/ea0d0c6e286b15d6eda70bdbdfc2a8c67d7e895d
Submitter: "Zuul (22348)"
Branch: stable/zed

commit ea0d0c6e286b15d6eda70bdbdfc2a8c67d7e895d
Author: Riccardo Pittau <email address hidden>
Date: Fri Dec 15 15:55:00 2023 +0100

    Handle LLDP parse Unicode error

    Closes-Bug: #2044793
    Change-Id: I939c80bb309b22e05ba7cc93686f12bbe0d71624
    (cherry picked from commit db76af9c86bffb61f9b7a52a4080959fee1658fa)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 11.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 11.1.2 release.