Nova boot fails if sbin not in path

[David McNally]

In a fresh install of devstack I see an error in the nova-compute log when I try to start an instance. I tracked this down to coming from nova/linux_net.py where sysctl is called in _enable_ipv4_forwarding().

If I add /sbin to my path the error goes away. However some distros, e.g.: debian, don't include sbin in the standard path rather they restrict it to root users only.

I think the call to sysctl (and possibly other similar calls) should be moved to use rootwrap, along with preventing issues like this it would have the added benifit of making the code slightly more OS agnostic.

[Loganathan Parthipan (parthipan)]

Also the use of sysctl to read as non-privileged user might not work with some security models.

[Loganathan Parthipan (parthipan)]

IMHO before implementation it would be good to step back take a look at the implications especially security and see if it is possible to use a python sysctl binding.

For instance, it's a good practice to do things as the least privileged user possible. If sysctl read is possible as non-root, please do so. That's what is currently done in Nova and the bug is that the error is handled wrong if executable is not found as dave-mcnally had found. All this could be refactored into a sysctl module so that it would do the right thing depending on the OS and security model.

[Angus Lees (gus)]

I figure we have these options, without a larger re-architect:

A. Invoke sysctl via sudo always
B. Invoke sysctl via a full path (/sbin/sysctl)
C. Add /usr/sbin:/sbin to PATH in nova
D. Require PATH to already include sysctl

D is the current situation and delaying while we discuss it further is a vote for D ;)
If we want to stay with this we need to fix PATH in devstack before invoking nova.

(I can see downsides with each option. I'm personally weakly in favour of A or D since they leave sysctl location up to the sysadmin/distro. I'm happy to do the patch for any of these, just want to know which choice is likely to be accepted.)

[Davanum Srinivas (DIMS) (dims-v)]

Hmm, we have a review for this but this bug did not get updated:
https://review.openstack.org/#/c/99820/

[Angus Lees (gus)]

I initially went with A for simplicity. Early review comments pushed for B. Later review comments quite conclusively pushed for D. So I'm abandoning the above change and writing a new change against devstack.

Review comments on https://review.openstack.org/#/c/99820/ for posterity:

Sean Dague:
I feel like we should just force a more extensive path. Or discover once somewhere where sysctl is. Working around path issues on distros seems weird.

Russell Bryant:
Agree with Sean. I don't think hardcoding the path makes a lot of sense. Perhaps this is something the debian openstack package maintainer can deal with to make sure sysctl is available to the user running nova.

Dan Prince:
If we follow this pattern here I feel like it might get replicated all over the place. Why not just fix the PATH (outside of nova) in the distro?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Angus Lees (<email address hidden>) on branch: master
Review: https://review.openstack.org/99820
Reason: Abandoning in favour of devstack fix to ensure sbin is in PATH before invoking nova:
https://review.openstack.org/108274

[Angus Lees (gus)]

Added openstack-manuals to find/fix anywhere where we talk about nova runtime requirements.

Wonderful docs people: Nova needs to be run with /sbin in PATH, so it can find sysctl. Presumably at least one other OpenStack service makes the same assumption, so it might be reasonable/easier to describe it as a requirement for all OpenStack services. If we talk about that anywhere, it should be updated to ensure operators are aware of this.

[Robbie Harwood (rharwood)]

See also https://bugs.launchpad.net/devstack/+bug/1207227

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/108274
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=7df9d1be17162feabeaba35faa87baf09debe590
Submitter: Jenkins
Branch: master

commit 7df9d1be17162feabeaba35faa87baf09debe590
Author: Angus Lees <email address hidden>
Date: Mon Jul 21 15:35:34 2014 +1000

    Ensure sbin is in PATH.

    Some distros (Debian) don't have sbin in PATH for non-root users.

    Nova (and possibly other services) assumes that it can invoke "sysctl"
    without sudo.

    Change-Id: Iced21fc1378af309fb49688f9b63f2cd8383e304
    Closes-Bug: #1300800

[Tom Fifield (fifieldt)]

This is not an issue in the install guide - the packages appear to set this up correctly.

[Tom Fifield (fifieldt)]

Thomas, can you have a quick look to see if this is an issue in debian?

[Tom Fifield (fifieldt)]

(I don't think it is, but since debian was specifically mentioned, just thought it was worth checking)

[Thomas Goirand (thomas-goirand)]

Hi. As much as I can tell, this isn't an issue in Debian. At least, I never noticed it.

[Tom Fifield (fifieldt)]

Thanks Thomas, marking as invalid for manuals.