Wrong haproxy-status on l3 agent node after restart

[Alexey. Kalashnikov]

Detailed bug description:
 During swarm tests one of them failed with ostf check error with haproxy status.
After reset controller with l3-agent, we got wrong status of backends not restarted nodes in out of haproxy-status.sh on restarted node

Seems like after reset, controller with l3 agent can't get status of other controllers and don't update them later

restarted node:
http://paste.openstack.org/show/545143/

other controllers:
http://paste.openstack.org/show/545144/

Steps to reproduce:
   1. Revert snapshot with neutron cluster
   2. Create an instance with a key pair
   3. Manually reschedule router from primary controller
           to another one
   4. Reset controller with l3-agent
   5. Check l3-agent was rescheduled
   6. Check network connectivity from instance via
           dhcp namespace
   7. Run OSTF

Expected results:
 all steps passed

Actual result:
OSTF check fialed with message:
http://paste.openstack.org/show/545145/

Description of the environment:
Snapshot #76

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vxlan/13/parameters/

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vxlan/13/console

logs:
https://drive.google.com/open?id=0B0EB6QSDWt2veUJGQXNSVmp3eGM

[Kyrylo Galanov (kgalanov)]

Unfortunately it is haproxy bug. Status is UP via http and DOWN via unix socket:
 echo "show info;show stat;show table" | socat /var/lib/haproxy/stats stdio
  --> http://paste.openstack.org/show/550407/

 curl "http://10.109.11.9:10000/;csv"
  --> http://paste.openstack.org/show/550409/

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/24647

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/24673

[Alexey. Kalashnikov (akalashnikov)]

Check on custom build os ok:
https://custom-ci.infra.mirantis.net/view/9.0/job/9.0.custom.system_test/1061/

[Timur Nurlygayanov (tnurlygayanov)]

The results of job https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vxlan/ are green, status changed to Fix Released for MOS 9.1