exit_trap fails trying to kill root processes when VERBOSE=True

[Thomas Morin]

I have a case of a jenkins job for which devstack.sh exit with 0 (success) with VERBOSE=False, but exits with 2 if VERBOSE is True even though no particular error is obvious (stack.sh finishes and output total times, keystone URL,users,password, etc.).

https://review.openstack.org/#/c/297260/30
with VERBOSE=False in local.conf
Works fine, we have err status 0:
http://logs.openstack.org/60/297260/30/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/ef556f6/console.html#_2016-04-06_19_10_42_040
2016-04-06 19:10:42.040 | gate_hook.sh: devstack-vm-gate.sh returned: 0

https://review.openstack.org/#/c/297260/29
only difference is that we don't have VERBOSE=False in local.conf
err status is now 2 and the job is marked as fail:
http://logs.openstack.org/60/297260/29/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/ca736ac/console.html#_2016-04-06_17_33_25_788
2016-04-06 17:33:25.788 | gate_hook.sh: devstack-vm-gate.sh returned: 2

[Ian Wienand (iwienand)]

Well this is an odd one

In [1] you can see

---
exit_trap: cleaning up child processes
./stack.sh: line 480: kill: (23227) - Operation not permitted+ unset GREP_OPTIONS
---

which is trying to kill from [2]

---
root 1 23227 0.0 0.0 40852 1752 ? S 17:33:22 00:00:00 sudo bagpipe-bgp start --no-daemon --log-file=/etc/bagpipe-bgp/log.conf
---

which is started at [3]

This kill only happens when we are running in VERBOSE mode [4]

Have to think about this ... generally we do not run things with sudo like this because they don't need it, they use tools like rootwrap to do privileged things

[1] http://logs.openstack.org/60/297260/29/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/ca736ac/logs/devstack-early.txt.gz
[2] http://logs.openstack.org/60/297260/29/experimental/gate-install-dsvm-networking-bgpvpn-bagpipe/ca736ac/logs/worlddump-2016-04-06-173325.txt.gz
[3] https://github.com/Orange-OpenSource/bagpipe-bgp/blob/master/devstack/plugin.sh#L116
[4] https://git.openstack.org/cgit/openstack-dev/devstack/tree/stack.sh#n476

[Thomas Morin (tmmorin-orange)]

Thanks a lot for getting to the root cause.

You are fully correct that it would be better to use rootwrap (or privilege separation) for bagpipe-bgp.
The project was initially designed to be usable outside an openstack environment as well, but there are many libraries including oslo_rootwrap that it would be fine to depend on. We will certainly consider this option.

[Thomas Morin (tmmorin-orange)]

networking-bagpipe now uses a version of bagpipe-bgp that uses oslo.rootwrap, which I expect to solve the issues