Nova scheduler is stopped after each reboot

[Radosław Piliszek]

Each reboot results in nova_scheduler container being down (properly stopped).

Simply deploy and reboot the node with nova_scheduler to reproduce it.

It does not seem to happen with docker restart.

Host: CentOS 7
Containers: CentOS 7 stein source

[Radosław Piliszek (yoctozepto)]

RestartPolicy is unless-stopped (correct)

There is seemingly nothing relevant in the logs of docker nor nova_scheduler (not running any debug atm).

[Laurent Dumont (baconpackets)]

I'm seeing the same behavior with Stein from source on Kolla 8.0.2

Nothing that seems of value in the container logs

2019-10-12 15:21:20.311 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 33 exited with status 0
2019-10-12 15:21:20.313 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 36 exited with status 0
2019-10-12 15:21:20.382 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 32 exited with status 0
2019-10-12 15:21:20.456 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 37 exited with status 0
2019-10-12 15:21:21.869 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 31 exited with status 0
2019-10-12 15:21:21.892 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 34 exited with status 0
2019-10-12 15:21:21.921 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 38 exited with status 0
2019-10-12 15:21:21.958 6 INFO oslo_service.service [req-4993422c-8481-49bf-ac05-f3b51f89d734 - - - - -] Child 35 exited with status 0

## When I manually restarted the container - docker start xxxx

2019-10-12 15:32:04.835 6 INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative
2019-10-12 15:32:05.371 6 INFO oslo_service.service [req-28f96a67-8ba8-4213-8963-73b52c4559c1 - - - - -] Starting 8 workers
2019-10-12 15:32:05.379 23 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.381 24 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.382 25 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.386 26 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.389 28 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.388 27 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.392 29 INFO nova.service [-] Starting scheduler node (version 19.0.2)
2019-10-12 15:32:05.401 30 INFO nova.service [-] Starting scheduler node (version 19.0.2)

There is this for the container stdout. It's not repeated when restarting and it might be shown during the shutdown.

Exception TypeError: "'NoneType' object is not callable" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:6> ignored

[Radosław Piliszek (yoctozepto)]

docker 19.03.2

[Will Szumski (willjs)]

Why don't we use a restart policy of always by the way? Is it because we manually send containers signals for a clean shutdown?

[Mark Goddard (mgoddard)]

I have a CentOS 7 host with docker 19.03.4, ubuntu/binary/master containers, and do not see this issue. Tried three times to reproduce. I'll try again with centos containers.

[Mark Goddard (mgoddard)]

Reproduced on the same host using centos/source/master images.

nova==20.0.0 (Train GA)

[Mark Goddard (mgoddard)]

Also reproduced on the same host using ubuntu/source/master images, nova==20.0.0.

[Mark Goddard (mgoddard)]

> Why don't we use a restart policy of always by the way? Is it because we manually send containers > signals for a clean shutdown?

It means containers can be stopped manually if necessary and Docker won't try to restart them.

[Mark Goddard (mgoddard)]

Argh, now I can't reproduce on either centos or ubuntu containers :(

[Mark Goddard (mgoddard)]

Got it! It's caused by a Docker issue, and our use of SIGHUP in ansible/roles/nova/tasks/refresh_scheduler_cell_cache.yml.

You can reproduce the issue as follows:

docker kill --signal HUP nova_scheduler
systemctl restart docker
docker ps -a | grep nova_scheduler

Docker assumes that the signal will stop the container, and marks it as not being restartable. However SIGHUP typically does not stop a process.

This is captured in Docker bug https://github.com/moby/moby/issues/11065. Their solution appears to be to use --stop-signal to define which signal should be used to stop the container. This does not appear to work in my testing however.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690982

[Will Szumski (willjs)]

>> Why don't we use a restart policy of always by the way? Is it because we manually send containers signals for a clean shutdown?

> It means containers can be stopped manually if necessary and Docker won't try to restart them.

Isn't that what always does? From, https://docs.docker.com/config/containers/start-containers-automatically/:

always: Always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted. (See the second bullet listed in restart policy details)

[Radosław Piliszek (yoctozepto)]

@Will,
If we ever want to be able to keep containers stopped on restarts (think docker upgrades or whatever that we finally coordinate), we need the current one.

@Mark,
Yeah, that explains why I got it from reboots and not daemon restarts - it had a high chance that I eventually rerun k-a between reboots. Thanks for debugging that.
Regarding fix - I don't think we should refrain from using SIGHUP there. We might want to keep the service working (this or another later). The bug is a bug but we can just signal the process without the docker's intervention.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690982
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=6bdf202658e08bb9f43ca50334587b05dc4bac03
Submitter: Zuul
Branch: master

commit 6bdf202658e08bb9f43ca50334587b05dc4bac03
Author: Mark Goddard <email address hidden>
Date: Thu Oct 24 15:01:42 2019 +0100

    Fix nova scheduler down after first docker restart

    Due to a Docker bug [1] we cannot use Docker to send
    SIGHUP to the container because it will mark it as
    stopped.
    This patch sends the signal directly to the process,
    bypassing Docker.

    'changed_when: false' is also removed from the
    relevant task as it definitely changes the state.
    In the future we could do the refresh only if
    there really is a need for another one.

    [1] https://github.com/moby/moby/issues/11065

    Change-Id: Ief73bbd24568d6941384ea3330ab45f11aa42d37
    Co-authored-by: Radosław Piliszek <email address hidden>
    Closes-Bug: #1845244

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/692212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/692213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692213
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d4734437b1d828013384cb20475981def569d6af
Submitter: Zuul
Branch: stable/rocky

commit d4734437b1d828013384cb20475981def569d6af
Author: Mark Goddard <email address hidden>
Date: Thu Oct 24 15:01:42 2019 +0100

    Fix nova scheduler down after first docker restart

    Due to a Docker bug [1] we cannot use Docker to send
    SIGHUP to the container because it will mark it as
    stopped.
    This patch sends the signal directly to the process,
    bypassing Docker.

    'changed_when: false' is also removed from the
    relevant task as it definitely changes the state.
    In the future we could do the refresh only if
    there really is a need for another one.

    [1] https://github.com/moby/moby/issues/11065

    Change-Id: Ief73bbd24568d6941384ea3330ab45f11aa42d37
    Co-authored-by: Radosław Piliszek <email address hidden>
    Closes-Bug: #1845244
    (cherry picked from commit 6bdf202658e08bb9f43ca50334587b05dc4bac03)
    (cherry picked from commit 2242fceb73abac54ef46acb41997bf81a77692a3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692212
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2242fceb73abac54ef46acb41997bf81a77692a3
Submitter: Zuul
Branch: stable/stein

commit 2242fceb73abac54ef46acb41997bf81a77692a3
Author: Mark Goddard <email address hidden>
Date: Thu Oct 24 15:01:42 2019 +0100

    Fix nova scheduler down after first docker restart

    Due to a Docker bug [1] we cannot use Docker to send
    SIGHUP to the container because it will mark it as
    stopped.
    This patch sends the signal directly to the process,
    bypassing Docker.

    'changed_when: false' is also removed from the
    relevant task as it definitely changes the state.
    In the future we could do the refresh only if
    there really is a need for another one.

    [1] https://github.com/moby/moby/issues/11065

    Change-Id: Ief73bbd24568d6941384ea3330ab45f11aa42d37
    Co-authored-by: Radosław Piliszek <email address hidden>
    Closes-Bug: #1845244
    (cherry picked from commit 6bdf202658e08bb9f43ca50334587b05dc4bac03)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.