[Backport bug #1511401] L3 agent no longer restarts on SIGHUP

[Kristina Berezovskaia]

On 8.0 we find the same errror: after sending SIGHUP signal to l3 agent, it doesn't reload config file and restart

Backport bug https://bugs.launchpad.net/neutron/+bug/1511401

----Original description-------------

l3-agent, as well as other neutron services is launched using oslo.service which sets handlers for such signals as SIGTERM and SIGHUP. On receiving SIGHUP a service is supposed to reload its config files and restart. [1]
Meanwhile, this is not so for l3 agent. PrefixDelegation class that is instantiated in L3NATAgent overrides handler for SIGHUP [2], thus removing handler that was set in oslo.service.

To restore the expected behavior self.pd_update_cb() should be moved to the reset method.

[1] http://docs.openstack.org/developer/oslo.service/usage.html#signal-handling
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/pd.py#L281-L288

___________________________________

Find on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "86"
  build_id: "86"
  fuel-nailgun_sha: "24b00023b4fc27885ffacda39696e29439c46040"
  python-fuelclient_sha: "2a1b048cc439986e222ece43a290b5cc68e92a77"
  fuel-agent_sha: "d2103bee6e216396eb8e308ec5448328c9ee4261"
  fuel-nailgun-agent_sha: "00b4b11553c250f22c0079fb74c8b782dcb7b740"
  astute_sha: "cfd5d6b916a17ad2f73e6c567a0365845155b0e3"
  fuel-library_sha: "8b72e8bd7d1a7d9c44f84ed5b1b479084eea6e49"
  fuel-ostf_sha: "9f500668555292add5d87c942e0cd804aefa6df2"
  fuel-createmirror_sha: "95cd15286fae4729d72d240b31ac7106c0222b5c"
  fuelmenu_sha: "5adac39504cd1b82ea3b7dc14b26c8cd93182239"
  shotgun_sha: "a63e74e80269c7c20188ee492f9b1ccfea72c6b1"
  fuelmain_sha: "03bdcf7ead2ea09e9493e580b0ee3288a931ae86"

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Elena Ezhova <email address hidden>
Review: https://review.fuel-infra.org/13535

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Elena Ezhova <email address hidden>
Review: https://review.fuel-infra.org/14041

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14041
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: fbbce6be9070b157897304ae3c46c04c043b881c
Author: Elena Ezhova <email address hidden>
Date: Fri Nov 20 11:30:45 2015

Refactor test_server functional tests

* Launching services with 0 workers makes no sense and would soon
  be prohibited in oslo.service. Passing workers=1 means that a
  service will be run in a current process.
* Extract the dummy _check_active method to the base class.

Related-Bug: #1512275
Cherry-picked from: https://review.openstack.org/#/c/245115/
Change-Id: Ib687bf1b321845098e0564ada46e68a7850b01c6

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13535
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: d6ff039e97f0afab3829e90e33952888ab49384a
Author: Elena Ezhova <email address hidden>
Date: Fri Nov 20 11:30:45 2015

Use SIGUSR1 to notify l3 agent of changing prefix file

It is common for all OpenStack services to use SIGHUP signal to
reload configuration and restart. This functionality, including
defining signal handler, is defined by oslo.service.

Meanwhile, this is currently not so for l3 agent. PrefixDelegation
class that is instantiated in L3NATAgent overrides handler for SIGHUP,
thus removing handler that was set in oslo.service.

The proposed solution is to use another signal, such as SIGUSR1,
instead of SIGHUP to notify l3 agent that a prefix file was
somehow changed.

Added a functional test for restarting L3 agent using SIGHUP.

Conflicts:
 neutron/tests/functional/agent/test_l3_agent.py

Closes-Bug: #1512275
Cherry-picked from https://review.openstack.org/#/c/240267/
Change-Id: I48eb4697a5fad97bcf08cfe1f80921a46d94029d

[Kristina Berezovskaia (kkuznetsova)]

Verify on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "232"
  build_id: "232"
  fuel-nailgun_sha: "0a2f99530cf7246acf85db643032a0550168aac5"
  python-fuelclient_sha: "b1ffe1cae9ce7b612d3f746c8e2e2fde6f732748"
  fuel-agent_sha: "bd67efbadabfd8242c979c50b7d61a251621621a"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "b60624ee2c5f1d6d805619b6c27965a973508da1"
  fuel-library_sha: "8b22d9db4d490cd9beb9261c15e0571bd3b3e7d6"
  fuel-ostf_sha: "a98973482f839554d90cc1c071d625a01e018cfe"
  fuel-createmirror_sha: "3cb98030d4a12992ea1cda1f464f035980569d2f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "25a0cc461a9fa4f7684f04cef0ff4ad9aa99a64d"
  network-checker_sha: "0b1b94a9685c6471d6911dff7ecac10b7bd2625f"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "2eca6adc33f02e02cd812e1d4be7c70e05fd07db"
(vxlan, 3 controllers, 2 compute)
Go the controller node and run script:
#!/bin/bash
. openrc
# Test SIGHUP signal
echo "Start testing SIGHUP signal on controller" > sighup_log
echo "
"

#
echo "__________________________________________________________________________
" >> sighup_log
screen -S l3_SIG -d -m -- sh -c 'tailf /var/log/neutron/l3-agent.log > log_l3' >> sighup_log
TEST_NAME="Restart l3-agent with kill SIGHUP command"
TEST_FAILED=0
echo "Start test "$TEST_NAME >> sighup_log
echo "Get a pid of a process for l3-agent" >> sighup_log
pid_before=$(ps -aux | grep -v root | grep neutron-l3-agent | awk '{print $2}')
echo "Pid for l3-agent process is "$pid_before >> sighup_log
echo "Kill a process with HUP " >> sighup_log
kill -SIGHUP $pid_before >> sighup_log
sleep 10
echo "Check health of l3-agent" >> sighup_log
neutron agent-list | grep 'L3 agent' >> sighup_log
echo "Сheck status of a process after HUP restar" >> sighup_log
pid_after=$(ps -aux | grep neutron-l3-agent | grep -v root | awk '{print $2}')
echo "PID after ="$pid_after >> sighup_log
if [ $pid_before = $pid_after ]; then echo "PIDs are equal" >> sighup_log; else echo "ERROR: pids are not equal"; TEST_FAILED=1; fi
screen -X -S l3_SIG kill
echo "Try to find ERROR in l3 log" >> sighup_log
cat log_l3 | grep ERROR >> sighup_log
echo "Try to find TRACE in l3 log" >> sighup_log
cat log_l3 | grep TRACE >> sighup_log
echo "Try to find SIGHUP command" >> sighup_log
cat log_l3 | grep SIGHUP >> sighup_log
NUMBER=$(cat log_l3 | grep SIGHUP | wc -l) >> sighup_log
if [ $NUMBER -ne 1 ]; then echo "ERROR: There was no SIGHUP command or it was sent more than 1 time" >> sighup_log; TEST_FAILED=1; fi
if [ $TEST_FAILED = 0 ]; then echo "Test "$TEST_NAME" PASSED" >> sighup_log; else echo "Test "$TEST_NAME" FAILED" >> sighup_log; fi
rm -r log_l3

Check sighup_log and see sending signal, no errors and traces