race condition in rootwrap daemon leaves processes behind when restarting the openvswitch-agent

[Ralf Haferkamp]

There is a race condition in the daemonization code of oslo.rootwrap that causes rootwrap-daemons to stay around when the parent process (e.g. neutron-openvswitch-agent) exits that the "wrong" point in time.

If neutron-openvswitch-agent is shutdown while it is just forking the neutron-rootwrap-daemon process, there is certain chance of it leaving a dangling neutron-rootwrap-daemon process behind. This happens when neutron-openvswitch-agent exits before it was able setup the Finalize() callback that is supposed to cleanup the forked rootwrap-daemon child on shutdown.

PS: In conjunction with https://bugs.launchpad.net/oslo.rootwrap/+bug/1658973 this prevents a successful restart of neutron-openvswitch-agent, because it fails to listen on port 6633.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/424642

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ralf Haferkamp (<email address hidden>) on branch: master
Review: https://review.openstack.org/424642

[Ralf Haferkamp (rhafer)]

I am a bit lost about how to fix this. Here's some more details about the race. This is from oslo_rootwrap/daemon.py:

111 try:
112 # allow everybody to connect to the socket
[..]
117 try:
118 # In Python 3 we have to use buffer to push in bytes directly
119 stdout = sys.stdout.buffer
120 except AttributeError:
121 stdout = sys.stdout
122 stdout.write(socket_path.encode('utf-8'))
123 stdout.write(b'\n')
124 stdout.write(bytes(server.authkey))

If the client (e.g. openvswitch-agent) exits before this, that's ok. We will notice as we receive a SIGPIPE. But now we're closing our communications channel with the client until the server.serve_forever() call (line 133) and until the client connected to it. If the client exit for whatever reason during this time, the rootwrap-daemon won't notice that will not stop. (Note: It's also not possible have the client kill the server as it's running as root, while the client is likely running as an unprivileged user.

125 sys.stdin.close()
126 sys.stdout.close()
127 sys.stderr.close()
128 # Gracefully shutdown on INT or TERM signals
129 stop = functools.partial(daemon_stop, server)
130 signal.signal(signal.SIGTERM, stop)
131 signal.signal(signal.SIGINT, stop)
132 LOG.info("Starting rootwrap daemon main loop")
133 server.serve_forever()
134 finally:

Any hints on how to fix this are highly appreciated.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438816
Committed: https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=6285b63572c893391cb1a9e0c482658938f13329
Submitter: Jenkins
Branch: master

commit 6285b63572c893391cb1a9e0c482658938f13329
Author: IWAMOTO Toshihiro <email address hidden>
Date: Tue Feb 28 15:12:01 2017 +0900

    Allow rootwrap-daemon to timeout and exit

    If the client side abnormally exits, its rootwrap daemon cannot
    receive a shutdown message and will be left forever. Let it timeout
    and exit to save such cases.

    Change-Id: I783717b5fa019371747b98bf92965b6e689603f6
    Related-bug: #1658973
    Related-bug: #1658977
    Related-bug: #1663458

[Adam Spiers (adam.spiers)]

Can this be marked as resolved now?

[Ralf Haferkamp (rhafer)]

Yeah seems the fix/workaround was released with oslo.rootwrap 5.6.0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/448744

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/448745

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448744
Committed: https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=fdacd0e60817db8455b3c2f21b60e8a2130953aa
Submitter: Jenkins
Branch: stable/ocata

commit fdacd0e60817db8455b3c2f21b60e8a2130953aa
Author: IWAMOTO Toshihiro <email address hidden>
Date: Tue Feb 28 15:12:01 2017 +0900

    Allow rootwrap-daemon to timeout and exit

    If the client side abnormally exits, its rootwrap daemon cannot
    receive a shutdown message and will be left forever. Let it timeout
    and exit to save such cases.

    Change-Id: I783717b5fa019371747b98bf92965b6e689603f6
    Related-bug: #1658973
    Related-bug: #1658977
    Related-bug: #1663458
    (cherry picked from commit 6285b63572c893391cb1a9e0c482658938f13329)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448745
Committed: https://git.openstack.org/cgit/openstack/oslo.rootwrap/commit/?id=af8ad2da809f68442da9aacd17a47bca342eb355
Submitter: Jenkins
Branch: stable/newton

commit af8ad2da809f68442da9aacd17a47bca342eb355
Author: IWAMOTO Toshihiro <email address hidden>
Date: Tue Feb 28 15:12:01 2017 +0900

    Allow rootwrap-daemon to timeout and exit

    If the client side abnormally exits, its rootwrap daemon cannot
    receive a shutdown message and will be left forever. Let it timeout
    and exit to save such cases.

    Change-Id: I783717b5fa019371747b98bf92965b6e689603f6
    Related-bug: #1658973
    Related-bug: #1658977
    Related-bug: #1663458
    (cherry picked from commit 6285b63572c893391cb1a9e0c482658938f13329)