cinderlib-lvm-functional job is failing

[Takashi Kajinami]

Currently the cinderlib-lvm-functional jobs in stable/ussuri is continuously failing.

Example:
https://zuul.opendev.org/t/openstack/build/12b30e492f35437680468b1747e88bb3

Looking at logs, the all failures are caused by the following import error.

ModuleNotFoundError: No module named 'os_brick.privileged.rootwrap'

[Takashi Kajinami (kajinamit)]

This issue was initially found in the following patch[1] which migrates the job from centos-8 to centos-8 stream.
 https://review.opendev.org/c/openstack/cinderlib/+/824421

The issue can be reproduced in centos-8 thus this issue is not specific to centos-8-steam.
 https://review.opendev.org/c/openstack/cinderlib/+/824939/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinderlib/+/847170

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinderlib/+/847170
Committed: https://opendev.org/openstack/cinderlib/commit/4d784d23a91f789918929ca397910f3121fa166e
Submitter: "Zuul (22348)"
Branch: master

commit 4d784d23a91f789918929ca397910f3121fa166e
Author: Gorka Eguileor <email address hidden>
Date: Wed Jun 22 14:58:55 2022 +0200

    Fix privsep issues inside virtual env

    When a virtual environment is created with the "--system-site-packages"
    option and privsep is installed on the system privsep will only use the
    system packages and completely ignore the ones in the virtual
    environment.

    This results in errors such as the ones we see:

    - In the Ussuri gate: ModuleNotFoundError: No module named
      'os_brick.privileged.rootwrap'

    - In the Wallaby gate: ModuleNotFoundError: No module named
      'os_brick.privileged.nvmeof'

    This happens because os-brick and cinder are starting privsep using the
    "privsep-helper" mechanism, and privsep was not installed in the virtual
    env because it was already present system wide, so the "privsep-helper"
    that is executed is the one from "/usr/local/bin/privsep-helper".

    This python script "privsep-helper" ignores the virtual environment and
    forces usage of the system's python, for example in a Wallaby
    installation this could be "#!/usr/bin/python3.6".

    Since it ignores the virtual environment it won't use its packages and
    anything that's not present on system wide will not be found, and if
    found it may be executing different code.

    This patch fixes this issue by replacing the helper used to start
    privsep with our own command.

    This command is the same as the one usually installed in /usr/local/bin
    but using /usr/bin/env to select the python to use.

    This new script has been included as data in the cinderlib namespace
    instead of making it install as a system script (like the original
    privsep command) because we don't want to polute the system wide
    binaries directory just for a corner case.

    We also need to preserve user site-packages for the running Python when
    calling root from the virtual environment, since the packages installed
    on the virtual environment with "--system-site-packages" would have
    taken those into consideration during the installation and not the ones
    present on the root user.

    To help debug issues at the gate all functional tests are now running
    with debug logs.

    Change-Id: I0278b42785d14f92a521e6deff872dcba6505270
    Related-Bug: #1958159
    Closes-Bug: #1979534

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinderlib/+/847622

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinderlib/+/847623

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinderlib/+/847622
Committed: https://opendev.org/openstack/cinderlib/commit/4fc56c815b0e9e4d3706dca14bbe54fbb4866272
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4fc56c815b0e9e4d3706dca14bbe54fbb4866272
Author: Gorka Eguileor <email address hidden>
Date: Wed Jun 22 14:58:55 2022 +0200

    Fix privsep issues inside virtual env

    When a virtual environment is created with the "--system-site-packages"
    option and privsep is installed on the system privsep will only use the
    system packages and completely ignore the ones in the virtual
    environment.

    This results in errors such as the ones we see:

    - In the Ussuri gate: ModuleNotFoundError: No module named
      'os_brick.privileged.rootwrap'

    - In the Wallaby gate: ModuleNotFoundError: No module named
      'os_brick.privileged.nvmeof'

    This happens because os-brick and cinder are starting privsep using the
    "privsep-helper" mechanism, and privsep was not installed in the virtual
    env because it was already present system wide, so the "privsep-helper"
    that is executed is the one from "/usr/local/bin/privsep-helper".

    This python script "privsep-helper" ignores the virtual environment and
    forces usage of the system's python, for example in a Wallaby
    installation this could be "#!/usr/bin/python3.6".

    Since it ignores the virtual environment it won't use its packages and
    anything that's not present on system wide will not be found, and if
    found it may be executing different code.

    This patch fixes this issue by replacing the helper used to start
    privsep with our own command.

    This command is the same as the one usually installed in /usr/local/bin
    but using /usr/bin/env to select the python to use.

    This new script has been included as data in the cinderlib namespace
    instead of making it install as a system script (like the original
    privsep command) because we don't want to polute the system wide
    binaries directory just for a corner case.

    We also need to preserve user site-packages for the running Python when
    calling root from the virtual environment, since the packages installed
    on the virtual environment with "--system-site-packages" would have
    taken those into consideration during the installation and not the ones
    present on the root user.

    To help debug issues at the gate all functional tests are now running
    with debug logs.

    Change-Id: I0278b42785d14f92a521e6deff872dcba6505270
    Related-Bug: #1958159
    Closes-Bug: #1979534
    (cherry picked from commit 4d784d23a91f789918929ca397910f3121fa166e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinderlib/+/847623
Committed: https://opendev.org/openstack/cinderlib/commit/1b58a5fb9af8d0beca964eee91f0e0040f21cb23
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 1b58a5fb9af8d0beca964eee91f0e0040f21cb23
Author: Gorka Eguileor <email address hidden>
Date: Wed Jun 22 14:58:55 2022 +0200

    Fix privsep issues inside virtual env

    When a virtual environment is created with the "--system-site-packages"
    option and privsep is installed on the system privsep will only use the
    system packages and completely ignore the ones in the virtual
    environment.

    This results in errors such as the ones we see:

    - In the Ussuri gate: ModuleNotFoundError: No module named
      'os_brick.privileged.rootwrap'

    - In the Wallaby gate: ModuleNotFoundError: No module named
      'os_brick.privileged.nvmeof'

    This happens because os-brick and cinder are starting privsep using the
    "privsep-helper" mechanism, and privsep was not installed in the virtual
    env because it was already present system wide, so the "privsep-helper"
    that is executed is the one from "/usr/local/bin/privsep-helper".

    This python script "privsep-helper" ignores the virtual environment and
    forces usage of the system's python, for example in a Wallaby
    installation this could be "#!/usr/bin/python3.6".

    Since it ignores the virtual environment it won't use its packages and
    anything that's not present on system wide will not be found, and if
    found it may be executing different code.

    This patch fixes this issue by replacing the helper used to start
    privsep with our own command.

    This command is the same as the one usually installed in /usr/local/bin
    but using /usr/bin/env to select the python to use.

    This new script has been included as data in the cinderlib namespace
    instead of making it install as a system script (like the original
    privsep command) because we don't want to polute the system wide
    binaries directory just for a corner case.

    We also need to preserve user site-packages for the running Python when
    calling root from the virtual environment, since the packages installed
    on the virtual environment with "--system-site-packages" would have
    taken those into consideration during the installation and not the ones
    present on the root user.

    To help debug issues at the gate all functional tests are now running
    with debug logs.

    Change-Id: I0278b42785d14f92a521e6deff872dcba6505270
    Related-Bug: #1958159
    Closes-Bug: #1979534
    (cherry picked from commit 4d784d23a91f789918929ca397910f3121fa166e)
    (cherry picked from commit 4fc56c815b0e9e4d3706dca14bbe54fbb4866272)

[Luigi Toscano (ltoscano)]

Shouldn't this be backported to ussuri too? But why is victoria apparently working?

[Brian Rosmaita (brian-rosmaita)]

We can't backport Change I0278b42785d14f92a521e6deff872dcba6505270 to victoria/ussuri/train because those branches are missing Change I7963fbfbb0a683e3efcc5949f80b96e5daaa18f1, which was intentionally not included in victoria [0]. That being said, I have no idea why this issue has not occurred in victoria.

[0] https://review.opendev.org/c/openstack/cinderlib/+/735935/12#message-9fd19925304d9c53f7b4cccffe689113c260e623

[Brian Rosmaita (brian-rosmaita)]

We won't fix this in stable/ussuri; it has been fixed through wallaby.