ironic-lib

Ironic inspection fails due to utf-8 decoding issue on Disk serial

Bug #2071935 reported by Riccardo Pittau
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ironic-lib
Fix Released
Low
Unassigned

Bug Description

This issue has been firstly reported as a bug downstream in Red Hat internal system.

The ironic-python-agent can't handle special characters in block device serial number.

Inspection fails during block devices identification if at least one of them contains special non utf-8 characters in the serial number.

Example from internal case:

Full stack trace:
~~~
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: 2024-07-03 09:16:11.628 1 DEBUG oslo_concurrency.processutils [-] CMD "lsblk -bia --json -oKNAME,MODEL,SIZE,ROTA,TYPE,UUID,PARTUUID,SERIAL" returned: 0 in 0.006s e
xecute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:422
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: --- Logging error ---
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: --- Logging error ---
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: Traceback (most recent call last):
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: File "/usr/lib64/python3.9/logging/__init__.py", line 1086, in emit
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: Traceback (most recent call last):
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib64/python3.9/logging/__init__.py", line 1086, in emit
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: stream.write(msg + self.terminator)
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1260-1267: surrogates not allowed
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: Call stack:
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: stream.write(msg + self.terminator)
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1260-1267: surrogates not allowed
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/bin/ironic-python-agent", line 10, in <module>
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: sys.exit(run())
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/cmd/agent.py", line 50, in run
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: agent.IronicPythonAgent(CONF.api_url,
Jul 03 09:16:11 master3.xxxxxx.yyy ironic-agent[2272]: Call stack:
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/agent.py", line 485, in run
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: self.process_lookup_data(content)
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/agent.py", line 400, in process_lookup_data
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: hardware.cache_node(self.node)
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 3179, in cache_node
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: dispatch_to_managers('wait_for_disks')
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 3124, in dispatch_to_managers
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: return getattr(manager, method)(*args, **kwargs)
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 997, in wait_for_disks
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: self.get_os_install_device()
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 1518, in get_os_install_device
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: block_devices = self.list_block_devices_check_skip_list(
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 1495, in list_block_devices_check_skip_list
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: block_devices = self.list_block_devices(
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 1460, in list_block_devices
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: block_devices = list_all_block_devices()
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_python_agent/hardware.py", line 526, in list_all_block_devices
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: report = il_utils.execute('lsblk', '-bia', '--json',
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_lib/utils.py", line 111, in execute
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: _log(result[0], result[1])
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: File "/usr/lib/python3.9/site-packages/ironic_lib/utils.py", line 99, in _log
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: LOG.debug('Command stdout is: "%s"', stdout)
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: Message: 'Command stdout is: "%s"'
Jul 03 09:16:11 master3.xxxxxx.yyy podman[2234]: Arguments: ('{\n "blockdevices": [\n {\n "kname": "loop0",\n "model": null,\n "size": 67467313152,\n "rota": false,\n "type": "loop",\n "uuid": "28f5ff52-7f5b-4e5a-bcf2-59813e5aef5a",\n "partuuid": null,\n "serial": null\n },{\n "kname": "loop1",\n "model": null,\n "size": 1027846144,\n "rota": false,\n "type": "loop",\n "uuid": null,\n "partuuid": null,\n "serial": null\n },{\n "kname": "sda",\n "model": "LITEON IT ECE-12",\n "size": 120034123776,\n "rota": false,\n "type": "disk",\n "uuid": null,\n "partuuid": null,\n "serial": "XXXXXXXXXXXXXXXXXX"\n },{\n "kname": "sdb",\n "model": "LITEON IT ECE-12",\n "size": 120034123776,\n "rota": false,\n "type": "disk",\n "uuid": null,\n "partuuid": null,\n "serial": "XXXXXXXXXXXXXXXXXXXX"\n },{\n "kname": "sdc",\n "model": "External",\n "size": 0,\n "rota": true,\n "type": "disk",\n "uuid": null,\n "partuuid": null,\n "serial": "2HC015KJ0000\udcff\udcff\udcff\udcff\udcff\udcff\udcff\udcff"\n }\n ]\n}\n',)

Tags: bug
Riccardo Pittau (rpittau)
tags: added: bug
Riccardo Pittau (rpittau)
Changed in ironic-python-agent:
importance: Undecided → Medium
Riccardo Pittau (rpittau)
affects: ironic-python-agent → ironic-lib
Changed in ironic-lib:
importance: Medium → Undecided
status: New → Triaged
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-lib (master)

Reviewed: https://review.opendev.org/c/openstack/ironic-lib/+/926045
Committed: https://opendev.org/openstack/ironic-lib/commit/1ca3c8cf1a90ccaed619df8b4ba97dbf9e386b98
Submitter: "Zuul (22348)"
Branch: master

commit 1ca3c8cf1a90ccaed619df8b4ba97dbf9e386b98
Author: Riccardo Pittau <email address hidden>
Date: Fri Aug 9 12:01:35 2024 +0200

    Fix invalid UTF-8 characters in execute output

    It's rare but the output of commands called by execute can
    contain invalid UTF-8 characters.
    While the execute command does not care about the output
    itself and will finish execution correctly, transposing
    that into string (for example when logging the output)
    will raise a Unicode exception.
    When that happens we should escape the invalid characters
    and log a trimmed output.

    Closes-Bug: 2071935
    Change-Id: I0c776e3fe5efa82b3220f573953bcac1678883f2

Changed in ironic-lib:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-lib (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/ironic-lib/+/926716

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-lib (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/ironic-lib/+/926718

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-lib (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/ironic-lib/+/926719

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic-lib 6.2.0

This issue was fixed in the openstack/ironic-lib 6.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-lib (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/ironic-lib/+/926716
Committed: https://opendev.org/openstack/ironic-lib/commit/aa4a2cd7da5821078ad417b98665073969d22115
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit aa4a2cd7da5821078ad417b98665073969d22115
Author: Riccardo Pittau <email address hidden>
Date: Fri Aug 9 12:01:35 2024 +0200

    Fix invalid UTF-8 characters in execute output

    It's rare but the output of commands called by execute can
    contain invalid UTF-8 characters.
    While the execute command does not care about the output
    itself and will finish execution correctly, transposing
    that into string (for example when logging the output)
    will raise a Unicode exception.
    When that happens we should escape the invalid characters
    and log a trimmed output.

    Closes-Bug: 2071935
    Change-Id: I0c776e3fe5efa82b3220f573953bcac1678883f2
    (cherry picked from commit 1ca3c8cf1a90ccaed619df8b4ba97dbf9e386b98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-lib (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/ironic-lib/+/926718
Committed: https://opendev.org/openstack/ironic-lib/commit/eff1475580fe8ba873fb12feb831a11b24b24709
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit eff1475580fe8ba873fb12feb831a11b24b24709
Author: Riccardo Pittau <email address hidden>
Date: Fri Aug 9 12:01:35 2024 +0200

    Fix invalid UTF-8 characters in execute output

    It's rare but the output of commands called by execute can
    contain invalid UTF-8 characters.
    While the execute command does not care about the output
    itself and will finish execution correctly, transposing
    that into string (for example when logging the output)
    will raise a Unicode exception.
    When that happens we should escape the invalid characters
    and log a trimmed output.

    Closes-Bug: 2071935
    Change-Id: I0c776e3fe5efa82b3220f573953bcac1678883f2
    (cherry picked from commit 1ca3c8cf1a90ccaed619df8b4ba97dbf9e386b98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-lib (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/ironic-lib/+/926719
Committed: https://opendev.org/openstack/ironic-lib/commit/e9cf477f9079b91f571b276e1c666a466828b0ec
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit e9cf477f9079b91f571b276e1c666a466828b0ec
Author: Riccardo Pittau <email address hidden>
Date: Fri Aug 9 12:01:35 2024 +0200

    Fix invalid UTF-8 characters in execute output

    It's rare but the output of commands called by execute can
    contain invalid UTF-8 characters.
    While the execute command does not care about the output
    itself and will finish execution correctly, transposing
    that into string (for example when logging the output)
    will raise a Unicode exception.
    When that happens we should escape the invalid characters
    and log a trimmed output.

    Closes-Bug: 2071935
    Change-Id: I0c776e3fe5efa82b3220f573953bcac1678883f2
    (cherry picked from commit 1ca3c8cf1a90ccaed619df8b4ba97dbf9e386b98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic-lib 5.4.1

This issue was fixed in the openstack/ironic-lib 5.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic-lib 5.5.1

This issue was fixed in the openstack/ironic-lib 5.5.1 Bobcat release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.