qemu-img calls should limit the memory usage

Bug #1657808 reported by Lucas Alvares Gomes on 2017-01-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ironic-lib
Fix Released
High
Lucas Alvares Gomes
ironic-python-agent
Fix Released
Undecided
Lucas Alvares Gomes

Bug Description

If you deploy many instances at the same time with the iSCSI driver the ironic-conductor can run out of memory suddenly because of the heavy memory usage from qemu-img {info, convert} calls. For example:

Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager 'qemu-img', 'info', path)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/ironic_lib/utils.py", line 90, in execute
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager result = processutils.execute(*cmd, **kwargs)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 363
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager env=env_variables)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 54, in
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager errread, errwrite)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager self.pid = os.fork()
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager OSError: [Errno 12] Cannot allocate memory
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager

Changed in ironic:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
importance: Undecided → High
no longer affects: ironic
Changed in ironic-lib:
importance: Undecided → High
assignee: nobody → Lucas Alvares Gomes (lucasagomes)

Fix proposed to branch: master
Review: https://review.openstack.org/422738

Changed in ironic-lib:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/422775

Changed in ironic-python-agent:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
status: New → In Progress

Reviewed: https://review.openstack.org/422738
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=4ae48d0b212c16c8b49d4f1144c073b3a3206597
Submitter: Jenkins
Branch: master

commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

    Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808

Changed in ironic-lib:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/422813
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=ace87b6eb39d7aab46301a1309b899c3635854e8
Submitter: Jenkins
Branch: stable/ocata

commit ace87b6eb39d7aab46301a1309b899c3635854e8
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

    Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)

tags: added: in-stable-ocata

This issue was fixed in the openstack/ironic-lib 2.5.2 release.

Reviewed: https://review.openstack.org/422854
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=152cf2857fd651d361c465887f35a035030042e2
Submitter: Jenkins
Branch: stable/newton

commit 152cf2857fd651d361c465887f35a035030042e2
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

    Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)

tags: added: in-stable-newton

Reviewed: https://review.openstack.org/422775
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=449cda4fed51250992446c83a4b50bc45fd10c6d
Submitter: Jenkins
Branch: master

commit 449cda4fed51250992446c83a4b50bc45fd10c6d
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

    Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

    Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
    Closes-Bug: #1657808

Changed in ironic-python-agent:
status: In Progress → Fix Released

This issue was fixed in the openstack/ironic-lib 2.1.3 release.

Reviewed: https://review.openstack.org/427134
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=ae0146151cd9b8c65af741eae4006cd46bdf6f21
Submitter: Jenkins
Branch: stable/newton

commit ae0146151cd9b8c65af741eae4006cd46bdf6f21
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

    Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

    Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
    Closes-Bug: #1657808
    (cherry picked from commit 449cda4fed51250992446c83a4b50bc45fd10c6d)

Reviewed: https://review.openstack.org/427135
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=60cd324ecd43d6c59e84d1f6a8a7735bee62f0fb
Submitter: Jenkins
Branch: stable/mitaka

commit 60cd324ecd43d6c59e84d1f6a8a7735bee62f0fb
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

    Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

    Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
    Closes-Bug: #1657808
    (cherry picked from commit 449cda4fed51250992446c83a4b50bc45fd10c6d)

tags: added: in-stable-mitaka

This issue was fixed in the openstack/ironic-python-agent 2.0.0 release.

This issue was fixed in the openstack/ironic-lib 2.6.0 release.

Reviewed: https://review.openstack.org/422855
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=ff4f92b3388d88a596ccaa45785376e507f03cd4
Submitter: Jenkins
Branch: stable/mitaka

commit ff4f92b3388d88a596ccaa45785376e507f03cd4
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

    Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)

This issue was fixed in the openstack/ironic-python-agent 1.5.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers