systemd based mount failures with Quobyte driver

Bug #1756823 reported by Silvan Kaiser on 2018-03-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Silvan Kaiser
Stein
Undecided
Silvan Kaiser

Bug Description

I'm rewriting the bug report as the original report was not accurate enough (still, the original report can be seen further down in this entry).

Updated bug report:
The Quobyte Nova driver checks for the presence of systemd in order to run mounts via systemd-run if available. However the systemd mount execution is broken in several aspects:
a) systemd detection has an additional whitespace in the path comparison, causing all instances to run without systemd
b) The systemd-run call is broken for CentOS as so far the currently installed systemd version on centos is 219 which does not support the '--user' flag
c) systemd had a bug for several releases that caused "systemctl-run --scope --user" calls to fail on e.g. Ubuntu Xenial (uses v229, fix in systemd was released in v238 according to [1]).
d) When run via processutils the systemd based mount command is executed in a context that does not allow using the "--user" flag resulting in an error as shown in the original bug report. This looks similar to executing systemd-run with --user option as user root.

For these reasons (b to d) the systemd mount variant in the driver needs to be run as root and the detection (a) to be fixed.

[1] https://github.com/systemd/systemd/blob/master/NEWS

Original bug report:
With different linux distros the systemd-run based mount of a Quobyte volume can fail. This is connected to the --user option causing issues with privilege escalation. Log example:

Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] ProcessExecutionError: Unexpected error while running command.
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Command: systemd-run --scope --user mount.quobyte --disable-xattrs 78.46.57.153:7861/
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Exit code: 1
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Stdout: u''
Mar 19 09:09:27 manualnovaci nova-compute[20516]: ERROR nova.compute.manager [instance: 6ffe17b6-9706-48d2-9ca5-73a670889a19] Stderr: u'Failed to create bus connection: No such file or directory\n'

Besides this the current systemd check is erroneous due to whitespace in path.

Silvan Kaiser (2-silvan) on 2018-03-19
Changed in nova:
assignee: nobody → Silvan Kaiser (2-silvan)

Fix proposed to branch: master
Review: https://review.openstack.org/554195

Changed in nova:
status: New → In Progress

The only bug I could find related to this was https://github.com/systemd/systemd/issues/3388. Is this one and the same (you should be able to see the logs in the journal)? If so, would relying on the distro to backport those fixes be a valid alternative to changing the behavior of the nova driver?

Silvan Kaiser (2-silvan) wrote :

Thanks for digging further. Although that bug is not what's hitting here afaics I also went back to dig into systemd and found the following:

1) systemd-run --scope --user mount.quobyte ... works fine on the cli without privileges
2) The same does not work when run via the Quobyte Nova driver, I think because of the oslo execution wrapping (will test this further tomorrow)
3) The error produced (Failed to create bus connection: No such file or directory) is easily reproducible by adding a sudo to the systemd-run cli command above.
4) I tested running the driver with explicitly setting run_as_root=True/False but the issue comes up in both variants.

So I'll take a closer look at the process execution tomorrow to find out what's causing the different behavior. Hints/help welcome. :)

Silvan Kaiser (2-silvan) wrote :

I rewrote the basic bug report (see original entry, original report is kept at the tail of that entry) in order to give a clearer overview on what's to be fixed.

description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/648093

Reviewed: https://review.openstack.org/554195
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7231f7dee10fa8f9e6cead026f6a5ae3f5b15ae4
Submitter: Zuul
Branch: master

commit 7231f7dee10fa8f9e6cead026f6a5ae3f5b15ae4
Author: Silvan Kaiser <email address hidden>
Date: Mon Mar 19 11:26:22 2018 +0100

    Exec systemd-run without --user flag in Quobyte driver

    Removes the --user flag from the systemd based mount command in the
    Quobyte driver. This prevents mount failures due to:
    - Older systemd releases not supporting the --user flag (e.g. CentOS)
    - Systemd versions having a bug preventing running the --scope and
      --user flags together
    - processutils context not allowing mount to run with this flag
    (see bug referenced below for details)

    Furthermore the systemd detection is fixed and
    all mount and umount commands are moved to libvirt privsep.

    Closes-Bug: #1756823

    Change-Id: I53f3a062ce419d1142d7dd3103fab565bb105e05

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/648093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7931f85fe66d4e024d7dd5396dcf38378af85470
Submitter: Zuul
Branch: master

commit 7931f85fe66d4e024d7dd5396dcf38378af85470
Author: Silvan Kaiser <email address hidden>
Date: Wed Mar 27 11:47:09 2019 +0100

    Adds systemd detection result caching in Quobyte driver

    Instead of rechecking for systemd multiple times check only
    once and cache the result in the Quobyte driver.

    Furthermore this fixes a range of minor nits like adding
    comments, generalizing the libvirt privsep umount and
    clarifying variable names in related tests, as noted
    in previous change I53f3a062ce419d1142d7dd3103fab565bb105e05 .

    Last but not least this adds unit tests for newly added
    functions in privsep.libvirt, using a fixture from
    change I53b8cca99729bcae6246c79f342f87f55a4ea95c .

    Related-bug: #1756823

    Change-Id: Iba30c49f108af9055de1b1a5c7b1a8406d66cf1a

Reviewed: https://review.opendev.org/649229
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=48f1826711cb286bace380a4e2468974df9a2f36
Submitter: Zuul
Branch: master

commit 48f1826711cb286bace380a4e2468974df9a2f36
Author: Artem Vasilyev <email address hidden>
Date: Tue Apr 2 09:13:00 2019 +0300

    systemd detection result caching nit fixes

    Change-Id: If54dd5c33a2e6535cc2ae9cf96179a202f6ed1a3
    Related-bug: #1756823

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.