neutron-linuxbridge-agent looping same as dhcp

[Rafał Radziejewski]

I am trying to install https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-ussuri on CentOS 8, with network provider option 1.

For the reproduction steps i followed install-guide deployment-for-ussuri.

Other components with trial's and errors did worked, neutron linuxbridge-agent keeps looping and returning errors.

Logs from linuxbridge-agent.log in the attachment additionally dhcp agent log returns similar problem in logs:

2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 action: enable call_driver /usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py:163
2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid; Error: [Errno 2] No such file or directory: '/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid' get_value_from_file /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:262
2020-07-10 12:21:15.061 68787 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/dhcp_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-dhcp-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmphzark8uo/privsep.sock']
2020-07-10 12:21:16.353 68787 CRITICAL oslo.privsep.daemon [-] privsep helper command exited non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 34dc4390-9448-4eba-8be2-a5c3f4cb94a5.: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 178, in call_driver
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 256, in enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 703, in wait_until_true
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent while not predicate():
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 268, in _enable
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1652, in setup
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent ip_lib.IPWrapper().ensure_namespace(network.namespace)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 249, in ensure_namespace
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent if not self.netns.exists(name):
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 728, in exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return network_namespace_exists(name)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 936, in network_namespace_exists
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent output = list_network_namespaces(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 922, in list_network_namespaces
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent return privileged.list_netns(**kwargs)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in _wrap
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent self.start()
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in start
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent channel = daemon.RootwrapClientChannel(context=self)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 331, in __init__
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent raise FailedToDropPrivileges(msg)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent

It tries to get /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid but it's not there

[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# pwd
/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5
[root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# ls -a
. ..

nmcli con show
NAME UUID TYPE DEVICE
eno3 ad51daed-db15-4380-b559-ccda28fec65e ethernet eno3
eno4 f4d476b2-edaf-45e6-bbb5-84d97b6b5e2f ethernet eno4

 openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+----------+--------------------------------------+
| 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 | provider | 26401d1b-38c4-4887-bef3-64b746f86e96 |
+--------------------------------------+----------+--------------------------------------+
 neutron agent-list

neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
| 1ff1b3be-555c-4ef2-a004-b1299f2a8d3d | DHCP agent | X | nova | :-) | True | neutron-dhcp-agent |
| ff6f0baf-e507-4df2-8cc3-a3eb4bc064e0 | Metadata agent | X | | :-) | True | neutron-metadata-agent |
+--------------------------------------+----------------+----

eno4 is interface used for provider it has'nt any ip address attached.

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1
sysctl net.bridge.bridge-nf-call-ip6tables net.bridge.bridge-nf-call-ip6tables = 1

journalctl -u neutron-linuxbridge-agent.service -xe --no-pager

Jul 10 12:31:02 X sudo[87036]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmpj1_5u88q/privsep.sock
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 CRITICAL privsep [-] Unhandled error: PermissionError: [Errno 13] Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep Traceback (most recent call last):
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep File "/bin/privsep-helper", line 10, in <module>
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep sys.exit(helper_main())
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 536, in helper_main
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep sock.connect(cfg.CONF.privsep_sock_path)
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep PermissionError: [Errno 13] Permission denied
Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Main process exited, code=exited, status=1/FAILURE
Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.
Jul 10 12:31:04 X systemd[1]: neutron-linuxbridge-agent.service: Service RestartSec=100ms expired, scheduling restart

cat /etc/sudoers.d/neutron
Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
neutron ALL = (root) NOPASSWD: ALL

In sudoers file there is directive #includedir /etc/sudoers.d

OpenStack version: Ussuri
Linux distro: CentOS 8

I am trying to install this controller/compute node on the same bare metal machine.

[Rafał Radziejewski (rafrad1994)]

UPDATE :

2020-07-10 13:52:58.511 136577 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qdhcp-34dc4390-9448-4eba-8be2-a5c3f4cb94a5": No such file or directory

2020-07-10 13:52:58.512 136577 WARNING neutron.agent.dhcp.agent [-] Network 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 may have been deleted and its resources may have already been disposed.: neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qdhcp-34dc4390-9448-4eba-8be2-a5c3f4cb94a5": No such file or directory

 openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+----------+--------------------------------------+
| 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 | provider | 26401d1b-38c4-4887-bef3-64b746f86e96 |

[Maciej Gravestone (guaz)]

Update:

Hi, I trying with Rafał get this openstack.

About this error:
```
ERROR neutron.agent.dhcp.agent oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
```
I get rid of this. Simply adding neutron to /etc/sudoers file:
```
neutron ALL=(ALL) NOPASSWD: ALL
```
and setup configuration for privsep in /etc/neutron/neutron.conf file:
```
[privsep]
user = root
helper_command = true
```

Idk if this is the best way to get rid of this, but after 'silent' results in google about this, it could be the best you'll find when this error raises.

[Maciej Gravestone (guaz)]

PS: Other errors still remains unsolved in our environment for now.

[Brian Haley (brian-haley)]

Are you using the RDO packages?

https://docs.openstack.org/install-guide/environment-packages-rdo.html

This does look more like a config issue than something with the agent itself.

Another link that looked useful:

https://www.server-world.info/en/note?os=CentOS_8&p=openstack_ussuri2&f=4

[Rafał Radziejewski (rafrad1994)]

Feel free to close this we cannot evaluate that anymore, thanks for willing to help.