haproxy container can not be stopped/restarted gracefully

Bug #1559238 reported by Jeffrey Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Jeffrey Zhang

Bug Description

in current master code [0], the haproxy container can not be stopped or restarted gracefully, after ran `kolla_ensure_haproxy_latest_config`

here is the reproduce step

1. deploy the kolla as normal
2. stop the haproxy,

$ time docker stop haproxy
haproxy

real 0m10.075s
user 0m0.027s
sys 0m0.009s

you can see the it takes 10 seconds to stop haproxy

check the docker daemon's log, a warning is raised.

Mar 18 13:48:25 localhost docker[915]: time="2016-03-18T13:48:25.440287036-04:00" level=info msg="Container 70ae72424501d028cc2129f9dc1fff19b3bd41eabcfe0e7e76a9c5bc4210c00b failed to exit within 10 seconds of SIGTERM - using the force"

The root cause is that, we can not use `exec` for haproxy, so the root process ( pid=1 ) is not haproxy,

$ docker exec haproxy ps auxf
root 1 2.0 0.0 11636 1440 ? Ss+ 17:49 0:00 /bin/bash /usr/local/bin/kolla_start
root 15 0.0 0.0 48800 2460 ? Ss 17:49 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

the running `docker stop haproxy`, the kolla_start(pid=1) will receive the signal, but haproxy process not. So the haproxy can not be stopped and will be killed after 10 seconds.

the best solution should be use `haproxy-systemd-wrapper` binary to start the haproxy process. this binary is original developed for systemd system case. But it is also good solution for docker. the official haproxy docker image use this also[1]

[0] https://github.com/openstack/kolla/tree/db0de9b7af52251ead2ddacf2a7c283691a6d6b2
[1] https://github.com/docker-library/haproxy/blob/a11db1597f9be5365028673df4d05b2ea854b3ed/1.6/docker-entrypoint.sh#L12

summary: - haproxy container can not be stop/restart gracefully
+ haproxy container can not be stopped/restarted gracefully
Steven Dake (sdake)
Changed in kolla:
status: New → Triaged
importance: Undecided → Medium
milestone: none → newton-1
assignee: nobody → Jeffrey Zhang (jeffrey4l)
Changed in kolla:
status: Triaged → Confirmed
Changed in kolla:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/294701
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=de6650874da1470f10d4561fd1e33829eaef3414
Submitter: Jenkins
Branch: master

commit de6650874da1470f10d4561fd1e33829eaef3414
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Sat Mar 19 00:24:44 2016 +0800

    Start the haproxy by using haproxy-system-wrapper

    haproxy-system-wrapper is a solution for systemd from upstream. it can
    handle the reload graceful.

    Change-Id: I6a3d141af065e429bd1be1b7252f5c6df1fda3bb
    Closes-Bug: #1559238

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 3.0.0.0b1

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.