ironic-lib

qemu-img calls should limit the memory usage

Bug #1657808 reported by Lucas Alvares Gomes on 2017-01-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	ironic-lib	Fix Released	High	Lucas Alvares Gomes
	ironic-python-agent	Fix Released	Undecided	Lucas Alvares Gomes

Bug Description

If you deploy many instances at the same time with the iSCSI driver the ironic-conductor can run out of memory suddenly because of the heavy memory usage from qemu-img {info, convert} calls. For example:

Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager 'qemu-img', 'info', path)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/ironic_lib/utils.py", line 90, in execute
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager result = processutils.execute(*cmd, **kwargs)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 363
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager env=env_variables)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 54, in
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager errread, errwrite)
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager self.pid = os.fork()
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager OSError: [Errno 12] Cannot allocate memory
Jan 18 20:24:07 controller-0.localdomain ironic-conductor[161689]: 2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager

Tags:

Lucas Alvares Gomes (lucasagomes) on 2017-01-19

Changed in ironic:
assignee:	nobody → Lucas Alvares Gomes (lucasagomes)
importance:	Undecided → High
no longer affects:	ironic
Changed in ironic-lib:
importance:	Undecided → High
assignee:	nobody → Lucas Alvares Gomes (lucasagomes)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix proposed to ironic-lib (master)

Fix proposed to branch: master
Review: https://review.openstack.org/422738

Changed in ironic-lib:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix proposed to ironic-python-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/422775

Changed in ironic-python-agent:
assignee:	nobody → Lucas Alvares Gomes (lucasagomes)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix proposed to ironic-lib (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/422813

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix merged to ironic-lib (master)

Reviewed: https://review.openstack.org/422738
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=4ae48d0b212c16c8b49d4f1144c073b3a3206597
Submitter: Jenkins
Branch: master

commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
Closes-Bug: #1657808

Changed in ironic-lib:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix merged to ironic-lib (stable/ocata)

Reviewed: https://review.openstack.org/422813
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=ace87b6eb39d7aab46301a1309b899c3635854e8
Submitter: Jenkins
Branch: stable/ocata

commit ace87b6eb39d7aab46301a1309b899c3635854e8
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix proposed to ironic-lib (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/422854

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix proposed to ironic-lib (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422855

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix included in openstack/ironic-lib 2.5.2

This issue was fixed in the openstack/ironic-lib 2.5.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-19: Fix merged to ironic-lib (stable/newton)

Reviewed: https://review.openstack.org/422854
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=152cf2857fd651d361c465887f35a035030042e2
Submitter: Jenkins
Branch: stable/newton

commit 152cf2857fd651d361c465887f35a035030042e2
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)

tags:

added: in-stable-newton

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-20: Fix merged to ironic-python-agent (master)

#10

Reviewed: https://review.openstack.org/422775
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=449cda4fed51250992446c83a4b50bc45fd10c6d
Submitter: Jenkins
Branch: master

commit 449cda4fed51250992446c83a4b50bc45fd10c6d
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
Closes-Bug: #1657808

Changed in ironic-python-agent:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-31: Fix included in openstack/ironic-lib 2.1.3

#11

This issue was fixed in the openstack/ironic-lib 2.1.3 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-31: Fix proposed to ironic-python-agent (stable/newton)

#12

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/427134

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-31: Fix proposed to ironic-python-agent (stable/mitaka)

#13

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/427135

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-01: Fix merged to ironic-python-agent (stable/newton)

#14

Reviewed: https://review.openstack.org/427134
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=ae0146151cd9b8c65af741eae4006cd46bdf6f21
Submitter: Jenkins
Branch: stable/newton

commit ae0146151cd9b8c65af741eae4006cd46bdf6f21
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

    Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
    Closes-Bug: #1657808
    (cherry picked from commit 449cda4fed51250992446c83a4b50bc45fd10c6d)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-02: Fix merged to ironic-python-agent (stable/mitaka)

#15

Reviewed: https://review.openstack.org/427135
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=60cd324ecd43d6c59e84d1f6a8a7735bee62f0fb
Submitter: Jenkins
Branch: stable/mitaka

commit 60cd324ecd43d6c59e84d1f6a8a7735bee62f0fb
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 17:16:38 2017 +0000

Limit memory usage when running qemu-img convert

    This patch is limiting the memory address space to 1 GiB when invoking
    qemu-img convert. The qemu-img command can be exploited so we better
    play on the safe side (See: bug #1449062).

    Change-Id: I2088f74b32d1392310c569cfaa7fd368cb737421
    Closes-Bug: #1657808
    (cherry picked from commit 449cda4fed51250992446c83a4b50bc45fd10c6d)

tags:

added: in-stable-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-13: Fix included in openstack/ironic-python-agent 2.0.0

#16

This issue was fixed in the openstack/ironic-python-agent 2.0.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-28: Fix included in openstack/ironic-lib 2.6.0

#17

This issue was fixed in the openstack/ironic-lib 2.6.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-24: Fix merged to ironic-lib (stable/mitaka)

#18

Reviewed: https://review.openstack.org/422855
Committed: https://git.openstack.org/cgit/openstack/ironic-lib/commit/?id=ff4f92b3388d88a596ccaa45785376e507f03cd4
Submitter: Jenkins
Branch: stable/mitaka

commit ff4f92b3388d88a596ccaa45785376e507f03cd4
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu Jan 19 16:22:30 2017 +0000

Limit memory usage when running qemu-img

    With multiple qemu-img processes running at the same time the host machine
    can quickly run out of memory. This patch is limiting the memory space
    to 1 GiB when invoking qemu-img.

    Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
    Closes-Bug: #1657808
    (cherry picked from commit 4ae48d0b212c16c8b49d4f1144c073b3a3206597)