systemd based mount failures with Quobyte driver

[Silvan Kaiser]

I'm rewriting the bug report as the original report was not accurate enough (still, the original report can be seen further down in this entry).

Updated bug report:
The Quobyte Nova driver checks for the presence of systemd in order to run mounts via systemd-run if available. However the systemd mount execution is broken in several aspects:
a) systemd detection has an additional whitespace in the path comparison, causing all instances to run without systemd
b) The systemd-run call is broken for CentOS as so far the currently installed systemd version on centos is 219 which does not support the '--user' flag
c) systemd had a bug for several releases that caused "systemctl-run --scope --user" calls to fail on e.g. Ubuntu Xenial (uses v229, fix in systemd was released in v238 according to [1]).
d) When run via processutils the systemd based mount command is executed in a context that does not allow using the "--user" flag resulting in an error as shown in the original bug report. This looks similar to executing systemd-run with --user option as user root.

For these reasons (b to d) the systemd mount variant in the driver needs to be run as root and the detection (a) to be fixed.

[1] https://github.com/systemd/systemd/blob/master/NEWS

Original bug report:
With different linux distros the systemd-run based mount of a Quobyte volume can fail. This is connected to the --user option causing issues with privilege escalation. Log example:

Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] ProcessExecutionError: Unexpected error while running command.
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Command: systemd-run --scope --user mount.quobyte --disable-xattrs 78.46.57.153:7861/
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Exit code: 1
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Stdout: u''
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Stderr: u'Failed to create bus connection: No such file or directory\n'

Besides this the current systemd check is erroneous due to whitespace in path.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/554195

[Stephen Finucane (stephenfinucane)]

The only bug I could find related to this was https://github.com/systemd/systemd/issues/3388. Is this one and the same (you should be able to see the logs in the journal)? If so, would relying on the distro to backport those fixes be a valid alternative to changing the behavior of the nova driver?

[Silvan Kaiser (2-silvan)]

Thanks for digging further. Although that bug is not what's hitting here afaics I also went back to dig into systemd and found the following:

1) systemd-run --scope --user mount.quobyte ... works fine on the cli without privileges
2) The same does not work when run via the Quobyte Nova driver, I think because of the oslo execution wrapping (will test this further tomorrow)
3) The error produced (Failed to create bus connection: No such file or directory) is easily reproducible by adding a sudo to the systemd-run cli command above.
4) I tested running the driver with explicitly setting run_as_root=True/False but the issue comes up in both variants.

So I'll take a closer look at the process execution tomorrow to find out what's causing the different behavior. Hints/help welcome. :)

[Silvan Kaiser (2-silvan)]

I rewrote the basic bug report (see original entry, original report is kept at the tail of that entry) in order to give a clearer overview on what's to be fixed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/648093

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554195
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7231f7dee10fa8f9e6cead026f6a5ae3f5b15ae4
Submitter: Zuul
Branch: master

commit 7231f7dee10fa8f9e6cead026f6a5ae3f5b15ae4
Author: Silvan Kaiser <email address hidden>
Date: Mon Mar 19 11:26:22 2018 +0100

    Exec systemd-run without --user flag in Quobyte driver

    Removes the --user flag from the systemd based mount command in the
    Quobyte driver. This prevents mount failures due to:
    - Older systemd releases not supporting the --user flag (e.g. CentOS)
    - Systemd versions having a bug preventing running the --scope and
      --user flags together
    - processutils context not allowing mount to run with this flag
    (see bug referenced below for details)

    Furthermore the systemd detection is fixed and
    all mount and umount commands are moved to libvirt privsep.

    Closes-Bug: #1756823

    Change-Id: I53f3a062ce419d1142d7dd3103fab565bb105e05

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7931f85fe66d4e024d7dd5396dcf38378af85470
Submitter: Zuul
Branch: master

commit 7931f85fe66d4e024d7dd5396dcf38378af85470
Author: Silvan Kaiser <email address hidden>
Date: Wed Mar 27 11:47:09 2019 +0100

    Adds systemd detection result caching in Quobyte driver

    Instead of rechecking for systemd multiple times check only
    once and cache the result in the Quobyte driver.

    Furthermore this fixes a range of minor nits like adding
    comments, generalizing the libvirt privsep umount and
    clarifying variable names in related tests, as noted
    in previous change I53f3a062ce419d1142d7dd3103fab565bb105e05 .

    Last but not least this adds unit tests for newly added
    functions in privsep.libvirt, using a fixture from
    change I53b8cca99729bcae6246c79f342f87f55a4ea95c .

    Related-bug: #1756823

    Change-Id: Iba30c49f108af9055de1b1a5c7b1a8406d66cf1a

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/649229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/649229
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=48f1826711cb286bace380a4e2468974df9a2f36
Submitter: Zuul
Branch: master

commit 48f1826711cb286bace380a4e2468974df9a2f36
Author: Artem Vasilyev <email address hidden>
Date: Tue Apr 2 09:13:00 2019 +0300

    systemd detection result caching nit fixes

    Change-Id: If54dd5c33a2e6535cc2ae9cf96179a202f6ed1a3
    Related-bug: #1756823

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/660705

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/660705
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.