[focal-wallaby] Services not running that should be: neutron-openvswitch-agent

[Aurelien Lourot]

When deploying focal-wallaby, the neutron-openvswitch-agent service doesn't start because one of its dependency services refuses to start:

systemd[1]: Starting OpenStack Neutron OVS cleanup...
sudo[190474]: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory
sudo[190474]: pam_unix(sudo:auth): conversation failed
sudo[190474]: pam_unix(sudo:auth): auth could not identify password for [neutron]
sudo[190474]: neutron : command not allowed ; TTY=unknown ; PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/privsep-helper --config-file /etc/neutron/neutron.conf --privsep_context neutron.privileged.>
systemd[1]: neutron-ovs-cleanup.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: neutron-ovs-cleanup.service: Failed with result 'exit-code'.
systemd[1]: Failed to start OpenStack Neutron OVS cleanup.

Indeed /etc/securetty got removed in focal [0][1]. Maybe Neutron just started doing something in Wallaby that requires this file?

This is visible in our OSCI test gate. [2][3]

[0] https://bugs.launchpad.net/ubuntu/+source/pam/+bug/1860826
[1] https://askubuntu.com/questions/1239503/ubuntu-20-04-and-20-10-etc-securetty-no-such-file-or-directory
[2] https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/778932
[3] https://openstack-ci-reports.ubuntu.com/artifacts/1a8/778932/5/check/migrate-ovn-focal-wallaby-dvr-snat/1a84d92/log/juju-status.zaza-3b6febd0f883.txt

[Aurelien Lourot (aurelien-lourot)]

Problem:

$ sudo systemctl restart neutron-ovs-cleanup # fails

Workaround:

$ neutron-ovs-cleanup # succeeds and seems to unblock something, because now...
$ sudo systemctl restart neutron-ovs-cleanup # succeeds
$ sudo systemctl restart neutron-openvswitch-agent # succeeds

[Corey Bryant (corey.bryant)]

Neutron is in the process of migrating from rootwrap to privsep. It doesn't look like all of privsep transition has occurred, and rootwrap hasn't been removed. We currently don't install etc/neutron/rootwrap.d/privsep.filters so I'll update the package to do that. I think that will solve this issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:18.0.0~rc2-0ubuntu2

---------------
neutron (2:18.0.0~rc2-0ubuntu2) hirsute; urgency=medium

  * d/neutron-common.install: Add missing rootwrap filter for privsep
    (LP: #1923453).

 -- Corey Bryant <email address hidden> Mon, 12 Apr 2021 10:57:31 -0400

[Corey Bryant (corey.bryant)]

The previous change didn't fix this. I'm working on debugging to figure out what the problem is. It seems to be limited to new neutron code in wallaby.

[Corey Bryant (corey.bryant)]

I think this is an upstream bug for neutron. It looks like neutron/cmd/ovs_cleanup.py is missing a call to agent_config.setup_privsep() in main():

diff --git a/neutron/cmd/ovs_cleanup.py b/neutron/cmd/ovs_cleanup.py
index c6290909a8..8e75317e08 100644
--- a/neutron/cmd/ovs_cleanup.py
+++ b/neutron/cmd/ovs_cleanup.py
@@ -58,6 +58,7 @@ def main():
     conf = setup_conf()
     conf()
     config.setup_logging()
+ agent_config.setup_privsep()
     do_main(conf)

The problem is the oslo.privsep library is not getting initialized. In other words, init() [1] is not getting called in oslo_privsep/priv_context.py, therefore _HELPER_COMMAND_PREFIX is not getting set to root_helper as defined in neutron.conf [2].

So we end up running:
Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--privsep_context', 'neutron.privileged.ovs_vsctl_cmd', '--privsep_sock_path', '/tmp/tmpvvymywvv/privsep.sock']

Where we should instead we should be running:
Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--privsep_context', 'neutron.privileged.ovs_vsctl_cmd', '--privsep_sock_path', '/tmp/tmpdq_rjxpi/privsep.sock']

[1] https://opendev.org/openstack/oslo.privsep/src/branch/stable/wallaby/oslo_privsep/priv_context.py#L107

[2] neutron.conf
[AGENT]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/786272

[Corey Bryant (corey.bryant)]

If this is the correct fix, neutron should get a wider audit to ensure all other commands are initializing the privsep library correctly.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Corey:

You are right, I'll check the sanity of other commands.

Regards.

[Corey Bryant (corey.bryant)]

Thanks Rodolfo. Pasting the other related bug here: https://bugs.launchpad.net/neutron/+bug/1923870

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:18.0.0-0ubuntu2

---------------
neutron (2:18.0.0-0ubuntu2) hirsute; urgency=medium

  * Cherry-pick patches from upstream master to initialize privsep library:
    - d/p/initialize-privsep-library-for-neutron-ovs-cleanup.patch (LP: #1923453).
    - d/p/initialize-privsep-library-in-neutron-commands.patch (LP: #1923870).

 -- Corey Bryant <email address hidden> Wed, 14 Apr 2021 13:13:40 -0400

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786272
Committed: https://opendev.org/openstack/neutron/commit/1dfbf5b3455391c849dc5ab7b61df86f6e0c8c16
Submitter: "Zuul (22348)"
Branch: master

commit 1dfbf5b3455391c849dc5ab7b61df86f6e0c8c16
Author: Corey Bryant <email address hidden>
Date: Wed Apr 14 11:45:47 2021 -0400

    Initialize privsep library for neutron-ovs-cleanup

    Closes-Bug: #1923453
    Change-Id: Ie91ab1c359955a2911e558ab5a4bc8b0ff28bada

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/787596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/787596
Committed: https://opendev.org/openstack/neutron/commit/85faee6cd4e206be57801dd8b2d52cb174035a3d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 85faee6cd4e206be57801dd8b2d52cb174035a3d
Author: Corey Bryant <email address hidden>
Date: Wed Apr 14 11:45:47 2021 -0400

    Initialize privsep library for neutron-ovs-cleanup

    Closes-Bug: #1923453
    Change-Id: Ie91ab1c359955a2911e558ab5a4bc8b0ff28bada
    (cherry picked from commit 1dfbf5b3455391c849dc5ab7b61df86f6e0c8c16)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.