Neutron agents have too many privsep-helper processes

[Darragh O'Reilly]

This is seen if the dhcp agent has more than 1 network to process when it starts, or if the l3-agent has more than one router to process when it starts. These agents process resources in concurrent threads and there seems to be a race creating the channel.

https://github.com/openstack/oslo.privsep/blob/00d599017844dc0580c8fbdebdb2b272628291b6/oslo_privsep/priv_context.py#L250-L261

$ ps -ef | grep privsep.*dhcp
root 2652 1 0 14:55 ? 00:00:00 /usr/bin/python3.6 /usr/local/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmpa5chbfm0/privsep.sock
root 2653 1 0 14:55 ? 00:00:00 /usr/bin/python3.6 /usr/local/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmp4uhfnadi/privsep.sock

The agent on this host processed 2 networks on startup:
$ ip netns | grep qdhcp
qdhcp-5773b11d-bbc1-4fc9-b0d1-7f640ccca717 (id: 1)
qdhcp-01a8f14e-5102-4428-8de4-8ec31694520e (id: 0)

The dhcp agent log has:

Feb 25 14:55:20 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
Feb 25 14:55:20 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] privsep daemon starting
Feb 25 14:55:20 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 2653
Feb 25 14:55:20 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
Feb 25 14:55:21 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] privsep daemon starting
Feb 25 14:55:21 ds3 neutron-dhcp-agent[929]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 2652

Only the last privsep daemon actually gets used.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/709764

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709764
Committed: https://git.openstack.org/cgit/openstack/oslo.privsep/commit/?id=bbe24aa6e08b253303a0f2332e92a7ce209c1afa
Submitter: Zuul
Branch: master

commit bbe24aa6e08b253303a0f2332e92a7ce209c1afa
Author: Darragh O'Reilly <email address hidden>
Date: Tue Feb 25 15:28:20 2020 +0000

    Add lock around channel creation

    When a neutron agent starts up, it processes different resources
    in separate eventlet threads. These can race creating the channel
    which results in redundant privsep-helper processes. This patch
    fixes that by adding a lock around channel creation.

    Change-Id: I5de22b72059133b05d64be47f4c1d3f566b46a6e
    Closes-Bug: #1864664

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.privsep 2.1.0 release.

[Peigan Xu (aganchongzi)]

When this issue will be fixed in the openstack/oslo.privsep stable/train branch？