the process in container can not handle the signal properly

Bug #1614509 reported by Jeffrey Zhang
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Jeffrey Zhang

Bug Description

When restarting containers, part of them can not handle the process properly. like when running `docker restart neutron_server`, you will get following warning:

Aug 18 08:26:07 aio dockerd[914]: time="2016-08-18T08:26:07.968485357-04:00" level=info msg="Container f614cdfad6a47b3a4821074da17d072408a6223f9fbd21016e6fa891a1dbaf48 failed to exit within 10 seconds of signal 15 - using the force"
Aug 18 08:26:08 aio dockerd[914]: time="2016-08-18T08:26:08.037156889-04:00" level=warning msg="libcontainerd: container f614cdfad6a47b3a4821074da17d072408a6223f9fbd21016e6fa891a1dbaf48 restart canceled"

it takes 10 seconds to stop the neutron_server with failed and send a 15 signal to the main process, which is bad.

The root cause is: The Linux kernel applies special signal handling to processes which run as PID 1. the solution is use the dumb-init as the init process in the container. more info please check: https://github.com/Yelp/dumb-init#why-you-need-an-init-system

Changed in kolla:
assignee: nobody → Jeffrey Zhang (jeffrey4l)
description: updated
Changed in kolla:
status: New → In Progress
Changed in kolla:
milestone: none → newton-3
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/357015
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=6710bbeb7c8bdcec45b1fb01c78618f6c9482e80
Submitter: Jenkins
Branch: master

commit 6710bbeb7c8bdcec45b1fb01c78618f6c9482e80
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Thu Aug 18 16:42:02 2016 +0800

    Use dumb-init to manage the pid 1 process

    Normally, when you launch a Docker container, the process you're
    executing becomes PID 1, giving it the quirks and responsibilities that
    come with being the init system for the container.

    There are two common issues this presents:

    * In most cases, signals won't be handled properly.
    * Orphaned zombie processes aren't properly reaped.

    the dumb-init acting like a simple init system. It launches a single
    process and then proxies all received signals to a session rooted at
    that child process.

    Closes-Bug: #1614509
    Change-Id: I9d3d04648e151ddc7c6732b92ffd3b6c9fe467ec

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 3.0.0.0b3

This issue was fixed in the openstack/kolla 3.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/364333

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/364333
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=7b365d77ebbbff6a43f6e1329e23a964a42be16f
Submitter: Jenkins
Branch: stable/mitaka

commit 7b365d77ebbbff6a43f6e1329e23a964a42be16f
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Thu Aug 18 16:42:02 2016 +0800

    Use dumb-init to manage the pid 1 process

    Normally, when you launch a Docker container, the process you're
    executing becomes PID 1, giving it the quirks and responsibilities that
    come with being the init system for the container.

    There are two common issues this presents:

    * In most cases, signals won't be handled properly.
    * Orphaned zombie processes aren't properly reaped.

    the dumb-init acting like a simple init system. It launches a single
    process and then proxies all received signals to a session rooted at
    that child process.

    Closes-Bug: #1614509
    Change-Id: I9d3d04648e151ddc7c6732b92ffd3b6c9fe467ec
    (cherry picked from commit 6710bbeb7c8bdcec45b1fb01c78618f6c9482e80)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 2.0.3

This issue was fixed in the openstack/kolla 2.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.