Unable to rootwrap kill some processes when running from a virtualenv

[Darren Worrall]

I'm running stable/juno in a virtualenv, and I'm getting errors like this when neutron's l3 agent tries to kill stale metadata agent processes:

/opt/openstack/venv/neutron/bin/neutron-rootwrap: Unauthorized command: kill -9 19387 (no filter matched)

This occurs because the absolute path of the python exe in question is /opt/openstack/venv/neutron/bin/python, but the last step in KillFilter.match only looks in the PATH environment variable for the executable[0]. Even though I can set PATH for the l3 agent daemon process via upstart or whatever, I think PATH gets reset to something more basic by sudo, because when I log what PATH is when the filter is called my additions are gone.

Could the filter also check the paths configured in exec_dirs for binaries?

[0]: https://github.com/openstack/oslo.rootwrap/blob/62d732277de5cf663172eafe6d3a6c149a9cf814/oslo/rootwrap/filters.py#L189-L192

[Darren Worrall (dazworrall)]

I was able to fix this myself by changing secure_path in the sudoers file:, eg:

Defaults:neutron !requiretty
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/openstack/venv/neutron/bin"

neutron ALL = (root) NOPASSWD: /opt/openstack/venv/neutron/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

I will still surprised that exec_dirs wasnt checked though, so I'll leave the bug open even if this is more of a documentation thing.

[Thierry Carrez (ttx)]

Thanks for filing this! Sounds like a reasonable request. I'll triplecheck the security implications though :)

[Thierry Carrez (ttx)]

Finally found the time to investigate this. Checking exec_dirs instead of the sudo-secured PATH would not cause security issues since exec_dirs is very much trusted too. That said, it is non-trivial to check exec_dirs as part of the match function of a filter (we need to change the match function arguments).

I consider this is a bit of a corner case (I don't think it can affect anything but killing 'python' processes ran from a venv), so I would rather document the limitation than change behavior to accommodate it.

But then, I'm not sure we should encourage switching secure_path (or exec_dirs) to list the venv bin directory. It is a security hole to specify (in exec_dirs or in secure_path) a directory that is not only writeable by root. I suspect in your case /opt/openstack/venv/neutron/bin is owned by some unprivileged user, and adding that to secure_path or exec_dirs may result in privilege escalation.

Which makes me ask, why does Neutron require root rights to kill a side Python process ? Will investigate more on Neutron side and report.

I also wonder WHY neutron needs root rights to kill this process -- like I said

[Darren Worrall (dazworrall)]

All the daemons and checkouts are owned by an unprivileged user, yes. Thanks for taking a look.

[Thierry Carrez (ttx)]

Ideally Neutron would just run neutron-ns-metadata-proxy as the neutron user and be able to kill it without requiring root rights. However Neutron starts neutron-ns-metadata-proxy as root, then drops to configurable metadata_proxy_user / metadata_proxy_group... hence the need to run the kill as root to be sure to kill it.

What looks like a security feature (configurable user) actually weakens the security (require allowing escalation to root to kill it)... not even mentioning corner cases like this bug.

Fixing it in Neutron would cause a backward incompatible change though, so I'd recommend the following workaround: add the following filter to your rootwrap filter definitions (probably in a file in /etc/neutron/rootwrap.d):

kill_venvmetadata: KillFilter, root, /opt/openstack/venv/neutron/bin/python, -9

That way you specifically allow this, without weakening your secure_path config.

[Thierry Carrez (ttx)]

Let me know if that works.