oslo.utils

qemu 4.1.0 output shifts to scientific notation at 1000mb, breaking oslo.utils (Advanced Virt 8.1.1)

Bug #1864529 reported by Herve Beraud
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.utils
Fix Released
Undecided
Herve Beraud

Bug Description

Problem here is that the qemu-img output shifts to scientific notation:

999 => 999 MiB
1000 => 1e+03 MiB

The regex in python-oslo-utils does not cover this.

This issue is likely regexp parsing "disk size: 1e+03 MiB" value:

$ qemu-img info overcloud-full.qcow2
image: overcloud-full.qcow2
file format: qcow2
virtual size: 3.84 GiB (4118413312 bytes)
disk size: 1e+03 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

$ ls -lha .
-rw-r--r--. 1 stack stack 1000M Feb 21 23:48 overcloud-full.qcow2

For comparison, different image (OSP16) gives:

$ qemu-img info 16/overcloud-full.qcow2
image: 16/overcloud-full.qcow2
file format: qcow2
virtual size: 3.84 GiB (4127129600 bytes)
disk size: 0.977 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

$ ls -lha . 16/
-rw-r--r--. 1 stack stack 1001M Feb 13 17:02 overcloud-full.qcow2

qemu-img-4.1.0-23.module+el8.1.1+5467+ba2d821b.x86_64
python-oslo-utils-lang-3.40.5-0.20200113183052.566e14c.el8ost.noarch

ironic-conductor on UC reports:

DEBUG ironic_lib.utils [req-741339a2-cccd-4ec6-9932-5d9e68af5d6f 6275ae0d0d904e868edeb26d56ae61d1 9c981643ec2f40578708640f060de538 - default default] Command stderr is: "" execute /usr/lib/python3.6/site-packages/
DEBUG oslo_concurrency.lockutils [req-741339a2-cccd-4ec6-9932-5d9e68af5d6f 6275ae0d0d904e868edeb26d56ae61d1 9c981643ec2f40578708640f060de538 - default default] Releasing lock "download-image" lock /usr/lib/python3
urrency/lockutils.py:281
ERROR ironic.conductor.utils [req-741339a2-cccd-4ec6-9932-5d9e68af5d6f 6275ae0d0d904e868edeb26d56ae61d1 9c981643ec2f40578708640f060de538 - default default] Node 95cdf531-ba45-444f-bc32-8d3acc35b624 failed deploy s
iority': 100, 'argsinfo': None, 'interface': 'deploy'} with unexpected error: Invalid string format: 1eB: ValueError: Invalid string format: 1eB
ERROR ironic.conductor.utils Traceback (most recent call last):
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/conductor/manager.py", line 3848, in _do_next_deploy_step
ERROR ironic.conductor.utils result = interface.execute_deploy_step(task, step)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/base.py", line 325, in execute_deploy_step
ERROR ironic.conductor.utils return self._execute_step(task, step)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/base.py", line 248, in _execute_step
ERROR ironic.conductor.utils return getattr(self, step['step'])(task)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
ERROR ironic.conductor.utils result = f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
ERROR ironic.conductor.utils return f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 422, in deploy
ERROR ironic.conductor.utils deploy_utils.cache_instance_image(task.context, node)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
ERROR ironic.conductor.utils result = f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/deploy_utils.py", line 1184, in cache_instance_image
ERROR ironic.conductor.utils force_raw)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/deploy_utils.py", line 521, in fetch_images
ERROR ironic.conductor.utils cache.fetch_image(href, path, ctx=ctx, force_raw=force_raw)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 140, in fetch_image
ERROR ironic.conductor.utils href, master_path, dest_path, ctx=ctx, force_raw=force_raw)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 167, in _download_image
ERROR ironic.conductor.utils _fetch(ctx, href, tmp_path, force_raw)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 321, in _fetch
ERROR ironic.conductor.utils required_space = images.converted_size(path_tmp)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic/common/images.py", line 392, in converted_size
ERROR ironic.conductor.utils data = disk_utils.qemu_img_info(path)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/ironic_lib/disk_utils.py", line 337, in qemu_img_info
ERROR ironic.conductor.utils return imageutils.QemuImgInfo(out)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/oslo_utils/imageutils.py", line 62, in __init__
ERROR ironic.conductor.utils details = self._parse(cmd_output or '')
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/oslo_utils/imageutils.py", line 178, in _parse
ERROR ironic.conductor.utils details = self._extract_details(root, root_details, lines)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/oslo_utils/imageutils.py", line 127, in _extract_details
ERROR ironic.conductor.utils real_details = self._extract_bytes(root_details)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/oslo_utils/imageutils.py", line 113, in _extract_bytes
ERROR ironic.conductor.utils return_int=True)
ERROR ironic.conductor.utils File "/usr/lib/python3.6/site-packages/oslo_utils/strutils.py", line 232, in string_to_bytes
ERROR ironic.conductor.utils raise ValueError(msg)
ERROR ironic.conductor.utils ValueError: Invalid string format: 1eB

Herve Beraud (herveberaud)
Changed in oslo.utils:
assignee: nobody → Herve Beraud (herveberaud)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.utils (master)

Fix proposed to branch: master
Review: https://review.opendev.org/709588

Changed in oslo.utils:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.utils (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/710016

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.utils (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/710017

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.utils (master)

Reviewed: https://review.opendev.org/709588
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=ebf8368501500767fd4e6e13602bb88950a4484b
Submitter: Zuul
Branch: master

commit ebf8368501500767fd4e6e13602bb88950a4484b
Author: Hervé Beraud <email address hidden>
Date: Mon Feb 24 20:15:48 2020 +0100

    Fix regex to correctly recognize scientific notation with QemuImgInfo

    qemu 4.1.0 output shifts to scientific notation at 1000mb, breaking
    oslo.utils.

    Problem here is that the qemu-img output shifts to scientific notation:

    999 => 999 MiB
    1000 => 1e+03 MiB

    The regex in python-oslo-utils does not cover this.

    This issue is likely regexp parsing "disk size: 1e+03 MiB" value.

    These changes fix that.

    Change-Id: I4c016865890135023ceb497de18d75ccebd5961a
    Closes-Bug: 1864529

Changed in oslo.utils:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.utils 4.1.0

This issue was fixed in the openstack/oslo.utils 4.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.utils (stable/train)

Reviewed: https://review.opendev.org/710016
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=85cd57d1c5b7f1e046f831b9b1caa341b688cfb4
Submitter: Zuul
Branch: stable/train

commit 85cd57d1c5b7f1e046f831b9b1caa341b688cfb4
Author: Hervé Beraud <email address hidden>
Date: Mon Feb 24 20:15:48 2020 +0100

    Fix regex to correctly recognize scientific notation with QemuImgInfo

    qemu 4.1.0 output shifts to scientific notation at 1000mb, breaking
    oslo.utils.

    Problem here is that the qemu-img output shifts to scientific notation:

    999 => 999 MiB
    1000 => 1e+03 MiB

    The regex in python-oslo-utils does not cover this.

    This issue is likely regexp parsing "disk size: 1e+03 MiB" value.

    These changes fix that.

    Change-Id: I4c016865890135023ceb497de18d75ccebd5961a
    Closes-Bug: 1864529
    (cherry picked from commit ebf8368501500767fd4e6e13602bb88950a4484b)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.utils (stable/stein)

Reviewed: https://review.opendev.org/710017
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=272786fd7efc50bede4824be9da685327dc5cc64
Submitter: Zuul
Branch: stable/stein

commit 272786fd7efc50bede4824be9da685327dc5cc64
Author: Hervé Beraud <email address hidden>
Date: Mon Feb 24 20:15:48 2020 +0100

    Fix regex to correctly recognize scientific notation with QemuImgInfo

    qemu 4.1.0 output shifts to scientific notation at 1000mb, breaking
    oslo.utils.

    Problem here is that the qemu-img output shifts to scientific notation:

    999 => 999 MiB
    1000 => 1e+03 MiB

    The regex in python-oslo-utils does not cover this.

    This issue is likely regexp parsing "disk size: 1e+03 MiB" value.

    These changes fix that.

    Change-Id: I4c016865890135023ceb497de18d75ccebd5961a
    Closes-Bug: 1864529
    (cherry picked from commit ebf8368501500767fd4e6e13602bb88950a4484b)
    (cherry picked from commit 85cd57d1c5b7f1e046f831b9b1caa341b688cfb4)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.utils 3.41.5

This issue was fixed in the openstack/oslo.utils 3.41.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.utils 3.40.6

This issue was fixed in the openstack/oslo.utils 3.40.6 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.