large performance impact of high fd limit on oslo.rootwrap

[Dirk Mueller]

When the default of filedescriptor ulimit for "root" is set to a high value and PTI ("meltdown") mitigations are enabled, the python 2.x "close_fds=True" code is excessively slow, because it effectively does something like:

for i in xrange(3, fd_limit):
  os.close(i)

leading to e.g. with the limit being 1 million to the equal amount of syscalls, which takes on medium/mid range reasonable hardware 400ms to process.

We could mitigate that by switching oslo.rootwrap for python 3 only, or removing close_fds or reimplement close_fds with slightly more intelligent code on linux (e.g. by traversing /proc/self/fd and looking for actually open fds instead of blindly closing all possible ones). or by just enforcing a low ulimit on fds and leaving the code as is.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607951
Committed: https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=c0a86998203315858721a7b2c8ab75fbf5cd51d9
Submitter: Zuul
Branch: master

commit c0a86998203315858721a7b2c8ab75fbf5cd51d9
Author: Dirk Mueller <email address hidden>
Date: Thu Oct 4 14:37:25 2018 +0200

    Run rootwrap with lower fd ulimit by default

    On Python 2.x, a subprocess.Popen() with close_fds=True will
    fork and then close filedescriptors range(3..os.sysconf("SC_OPEN_MAX")),
    which thanks to Kernel PTI (Kaiser patches) is significantly slower
    in 2018 when the range is very large. With a soft limit of 1048576,
    benchmark.py reports an overhead of ~ 400ms without this patch and 2ms
    with the patch applied. This patch adds a configuration option and
    sets a more sensible default of 1024 file descriptor limit by default.

    Closes-Bug: 1796267
    Change-Id: Idd98c183eca3e2df8648fc0f37d27fe9cc6d0563

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613387

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/613388

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/613389

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.rootwrap 5.15.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/613387
Committed: https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=fdb32096ab1fafbacf116b722140b24f0e2cd834
Submitter: Zuul
Branch: stable/rocky

commit fdb32096ab1fafbacf116b722140b24f0e2cd834
Author: Dirk Mueller <email address hidden>
Date: Thu Oct 4 14:37:25 2018 +0200

    Run rootwrap with lower fd ulimit by default

    On Python 2.x, a subprocess.Popen() with close_fds=True will
    fork and then close filedescriptors range(3..os.sysconf("SC_OPEN_MAX")),
    which thanks to Kernel PTI (Kaiser patches) is significantly slower
    in 2018 when the range is very large. With a soft limit of 1048576,
    benchmark.py reports an overhead of ~ 400ms without this patch and 2ms
    with the patch applied. This patch adds a configuration option and
    leaves it disabled for the stable/* backports to not change default
    behavior.

    Also includes Ben Nemec's release note entry, adjusted for the stable
    backport. This is Ib29e96307caa39c21936f216d9aed7907e7a7331 for master.

    Also includes I2391315f77718a3c9eb9fc8c03a6882237f33548 from master.

    Also includes I3b481ddd14ae2b948270d715aad157cf3996def7 from master.

    (cherry picked from commit c0a86998203315858721a7b2c8ab75fbf5cd51d9)
    (cherry picked from commit 8f148f5e9e786d8a3922b0d5ae29bc54b8e8e1e9)
    (cherry picked from commit 5259c08c09eb3aeab64af928cddf84bc213594d3)

    Closes-Bug: 1796267
    Closes-Bug: #1804639
    Change-Id: Idd98c183eca3e2df8648fc0f37d27fe9cc6d0563

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stephen Finucane (<email address hidden>) on branch: stable/pike
Review: https://review.opendev.org/613389
Reason: These branches are in extended maintenance, which means we won't be making new releases for them. Given that fact, along with the failing tests, I'm going to just abandon this

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stephen Finucane (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/613388
Reason: These branches are in extended maintenance, which means we won't be making new releases for them. Given that fact, along with the failing tests, I'm going to just abandon this

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.rootwrap 5.14.2 release.