neutron-l3-agent permission denied when execute ( list_network_namespaces)

[Hristo Ispirov]

I have neutron manual installation on ubuntu server 22.04.
I use 2023.1 openstack release.
Everything was working fine before I rebooted the machine.
Now I have this error:

2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task [None req-835bfb24-967b-44a4-849a-c661efe0c7e5 - - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task: PermissionError: [Errno 13] Permission denied
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task task(self, context)
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 887, in periodic_sync_routers_task
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task with self.namespaces_manager as ns_manager:
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/neutron/agent/l3/namespace_manager.py", line 71, in __enter__
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task self._all_namespaces = self.list_all()
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/neutron/agent/l3/namespace_manager.py", line 117, in list_all
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task namespaces = ip_lib.list_network_namespaces()
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 963, in list_network_namespaces
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task return privileged.list_netns(**kwargs)
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task return self.channel.remote_call(name, args, kwargs,
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task raise exc_type(*result[2])
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task PermissionError: [Errno 13] Permission denied
2023-07-05 12:20:55.449 48258 ERROR oslo_service.periodic_task

My /etc/sudoers.d/neutron_sudoers file contains :

Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

My /etc/neutron/rootwrap.d/rootwrap.filters file contains:

# Command filters to allow privsep daemon to be started via rootwrap.
#
# This file should be owned by (and only-writeable by) the root user

[Filters]

# By installing the following, the local admin is asserting that:
#
# 1. The python module load path used by privsep-helper
# command as root (as started by sudo/rootwrap) is trusted.
# 2. Any oslo.config files matching the --config-file
# arguments below are trusted.
# 3. Users allowed to run sudo/rootwrap with this configuration(*) are
# also allowed to invoke python "entrypoint" functions from
# --privsep_context with the additional (possibly root) privileges
# configured for that context.
#
# (*) ie: the user is allowed by /etc/sudoers to run rootwrap as root
#
# In particular, the oslo.config and python module path must not
# be writeable by the unprivileged user.

# PRIVSEP
# oslo.privsep default neutron context
privsep: PathFilter, privsep-helper, root,
 --config-file, /etc/(?!\.\.).*,
 --privsep_context, neutron.privileged.default,
 --privsep_sock_path, /

# NOTE: A second `--config-file` arg can also be added above. Since
# many neutron components are installed like that (eg: by devstack).
# Adjust to suit local requirements.

# DEBUG
sleep: RegExpFilter, sleep, root, sleep, \d+

# EXECUTE COMMANDS IN A NAMESPACE
ip: IpFilter, ip, root
ip_exec: IpNetnsExecFilter, ip, root

# METADATA PROXY
haproxy: RegExpFilter, haproxy, root, haproxy, -f, .*
haproxy_env: EnvFilter, env, root, PROCESS_TAG=, haproxy, -f, .*

# DHCP
dnsmasq: CommandFilter, dnsmasq, root
dnsmasq_env: EnvFilter, env, root, PROCESS_TAG=, dnsmasq

# DIBBLER
dibbler-client: CommandFilter, dibbler-client, root
dibbler-client_env: EnvFilter, env, root, PROCESS_TAG=, dibbler-client

# L3
radvd: CommandFilter, radvd, root
radvd_env: EnvFilter, env, root, PROCESS_TAG=, radvd
keepalived: CommandFilter, keepalived, root
keepalived_env: EnvFilter, env, root, PROCESS_TAG=, keepalived
keepalived_state_change: CommandFilter, neutron-keepalived-state-change, root
keepalived_state_change_env: EnvFilter, env, root, PROCESS_TAG=, neutron-keepalived-state-change

# OPEN VSWITCH
ovs-ofctl: CommandFilter, ovs-ofctl, root
ovsdb-client: CommandFilter, ovsdb-client, root

My /etc/neutron/rootwrap.conf file contains :
# Configuration for neutron-rootwrap
# This file should be owned by (and only-writeable by) the root user

[DEFAULT]
# List of directories to load filter definitions from (separated by ',').
# These directories MUST all be only writeable by root !
filters_path=/etc/neutron/rootwrap.d,/usr/share/neutron/rootwrap

# List of directories to search executables in, in case filters do not
# explicitely specify a full path (separated by ',')
# If not specified, defaults to system PATH environment variable.
# These directories MUST all be only writeable by root !
exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin,/etc/neutron/kill_scripts

# Enable logging to syslog
# Default value is False
use_syslog=False

# Which syslog facility to use.
# Valid values include auth, authpriv, syslog, local0, local1...
# Default value is 'syslog'
syslog_log_facility=syslog

# Which messages to log.
# INFO means log all usage
# ERROR means only log unsuccessful attempts
syslog_log_level=ERROR

# Rootwrap daemon exits after this seconds of inactivity
daemon_timeout=600

# Rootwrap daemon limits itself to that many file descriptors (Linux only)
rlimit_nofile=1024

In neutron config I have this :
[agent]
root_helper = "sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf"

I am not sure if it is neutron error or oslo.privsep error.

[Hristo Ispirov (hispirov)]

I found that if I add this section with capabilities :

[privsep_namespace]
capabilities = CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_SYS_PTRAC

in my /etc/neutron/neutron.conf file the error is gone

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Hristo:

When using a normal user (non-admin), to list the network namespaces no permission should be needed. But because some users have been capped, this is why we introduced the configuration variable "use_helper_for_ns_read", that is True by default. When enabled, the namespace listing [1] and the "path_exists" method [2] are wrapped inside a privsep context. But only "CAP_SYS_ADMIN" should be needed for these two operations.

I would suggest first to check the user permissions. With this user, can you execute "ip netns"? Can you read all the namespaces? You should be able.

Regards.

[1]https://github.com/openstack/neutron/blob/6b9333d8dfb78881a4368cb8716c48c45d49857f/neutron/agent/linux/ip_lib.py#L971
[2]https://github.com/openstack/neutron/blob/6b9333d8dfb78881a4368cb8716c48c45d49857f/neutron/agent/linux/ip_lib.py#L991

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

"Can you read all the namespaces?" is not accurate but "Can you list all the namespaces?"

[Hristo Ispirov (hispirov)]

Hello Rodolfo:

Yes I can execute "ip netns" with my root and neutron users with only this CAP_SYS_ADMIN capability but L3 Agent tries to call privileged.list_netns(**kwargs) and the execution is wrapped in privsep context and gets Permission denied Error.

In my configuration "use_helper_for_ns_read" property is "True".

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Hristo:

What does it mean "I can execute "ip netns" with my root and neutron users with only this CAP_SYS_ADMIN capability". How are you doing that? In a python console?

"privsep" daemon is a service running with the permissions provided; in this case "CAP_SYS_ADMIN". If you are able to execute "ip netns" with the user running the OpenStack services, then you can disable "use_helper_for_ns_read" because you don't need the "privsep" wrapper.

Regards.