Comment 7 for bug 1964111

Reviewed: https://review.opendev.org/c/starlingx/integ/+/834973
Committed: https://opendev.org/starlingx/integ/commit/169a0c0ee3416d0c082abeca59639cc479cf1866
Submitter: "Zuul (22348)"
Branch: master

commit 169a0c0ee3416d0c082abeca59639cc479cf1866
Author: Jim Gauld <email address hidden>
Date: Wed Mar 23 16:51:14 2022 -0400

    Move k8s container cleanup to containerd service

    This introduces k8s-container-cleanup script that will be called
    when containerd.service is stopped. The script detects whether systemd
    state is 'stopping' due to shutdown/reboot, then stops all running
    containers before the service shuts down.

    During shutdown/reboot, some containers are not receiving the
    SIGTERM signal. This leads to unexpected behaviour such as
    generating huge coredumps.

    There is an upstream issue regarding this:
    https://github.com/kubernetes/kubernetes/issues/107158
    The problem seems to be systemd related but this commit
    addresses the problem with a workaround.

    This reverts commit f3c18b0f79e3b145d378474b24d861926dd61a13.
    The k8s-container-cleanup script is moved from kubelet.service
    to containerd.service. The ExecStopPost that calls this script
    is removed, and replaced with ExecStop in containerd.service
    to call the script (in config-files repo).

    The k8s-container-cleanup script requires containerd is running
    in order to use crictl utility. The shutdown of kubelet and
    containerd have unpredictable timing, so the cleanup must be done
    in containerd.

    Test Plan: On AIO-SX
    PASS: Verify k8s-container-cleanup logs to daemon.log during 'stopping.
    PASS: Manual change containerd/kubelet shutdown timing and verify.
    k8s-container-cleanup running to completion before containerd stopped.
    PASS: Reboot and verify k8s-container-cleanup running to completion.
    PASS: Lock/unlock and verify k8s-container-cleanup running to completion.
    PASS: Manually run spellintian tool against k8s-container-cleanup.sh.
    PASS: Manually run shellcheck tool against k8s-container-cleanup.sh.
    PASS: Zuul tox bashate tool against k8s-container-cleanup.sh.

    Partial-Bug: 1964111
    Change-Id: Ic8a9e257f861ae218a8520205eced3eaa580dd20
    Signed-off-by: Jim Gauld <email address hidden>