restart of nova_virtlogd make the console.log to stop recording

[Martin Schuppert]

virtlogd daemon must *never* be stopped while there are running VMs. It is explicitly split out into a separate daemon from libvirtd so that it can upgrade its own software while keeping FDs open by re-exec'ing itself. If running virtlogd in a container this container must never be restarted while VMs are running.

We'd need to make virtlogd not be managed by paunch at all, and it should be treated in a similar way to the neutron l3[1] and dhcp[2] agents.

[1] https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/neutron/l3_agent_wrappers.pp
    https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/neutron/neutron-l3-container-puppet.yaml#L172-L186
[2] https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/neutron/dhcp_agent_wrappers.pp
    https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/neutron/neutron-dhcp-container-puppet.yaml

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/673307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673307
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=f6eb3c8db7f72acc54e80531b8f083ed084679f6
Submitter: Zuul
Branch: master

commit f6eb3c8db7f72acc54e80531b8f083ed084679f6
Author: Martin Schuppert <email address hidden>
Date: Mon Jul 29 16:59:21 2019 +0200

    Add docker/podman packages to nova_libvirt container

    The nova_libvirt containers need to be able to launch containers for
    the virtlogd process to be not managed by paunch.

    We only add docker if distro_python_version.startswith('2') . This
    applies to
    - nova_libvirt_packages_append
    - neutron_metadata_agent_ovn_packages_append
    - neutron_l3_agent_packages_append
    - neutron_dhcp_agent_packages

    Related-Bug: #1838272

    Change-Id: I5829addb715c4836dd27750daa67f2e472084b34

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/787771
Committed: https://opendev.org/openstack/puppet-tripleo/commit/f09aa48d95bc0fada01b1a011bc00e8c9b1e87b0
Submitter: "Zuul (22348)"
Branch: master

commit f09aa48d95bc0fada01b1a011bc00e8c9b1e87b0
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 23 16:52:45 2021 +0200

    Introduce nova virtlogd wrapper

    When nova_virtlogd container gets restarted the instance console auth files
    will not be reopened again by virtlogd. As a result either instances need
    to be restarted or live migrated to a different compute node to get new
    console logs messages logged again.
    Usually on receipt of SIGUSR1, virtlogd will re-exec() its binary, while
    maintaining all current logs and clients. This allows for live upgrades of
    the virtlogd service on non containerized environments where updates just
    by doing an RPM update.
    To reduce the likelihood in a containerized environment virtlogd should
    only be restarted on manual request, or on compute node reboot. It should
    not be restarted on a minor update without migration off instances.
    This introduces a nova_virtlogd_wrapper container and virtlogd wrapper
    script, to only restart virtlogd on either manual or compute node restart.

    Closes-Bug: #1838272

    Change-Id: I5192f8c306422d0966eef7544f719cc21172bd92
    Co-Authored-By: Rajesh Tailor <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/787772
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/49415d04b2f97583d63004420b500d454f62386a
Submitter: "Zuul (22348)"
Branch: master

commit 49415d04b2f97583d63004420b500d454f62386a
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 23 17:08:30 2021 +0200

    Introduce nova virtlogd wrapper

    When nova_virtlogd container gets restarted the instance console auth files
    will not be reopened again by virtlogd. As a result either instances need
    to be restarted or live migrated to a different compute node to get new
    console logs messages logged again.
    Usually on receipt of SIGUSR1, virtlogd will re-exec() its binary, while
    maintaining all current logs and clients. This allows for live upgrades of
    the virtlogd service on non containerized environments where updates just
    by doing an RPM update.
    To reduce the likelihood in a containerized environment virtlogd should
    only be restarted on manual request, or on compute node reboot. It should
    not be restarted on a minor update without migration off instances.
    This introduces a nova_virtlogd_wrapper container and virtlogd wrapper
    script, to only restart virtlogd on either manual or compute node restart.

    With NovaEnableVirtlogdContainerWrapper the virtlogd wrapper can be
    disabled.

    Co-Authored-By: Rajesh Tailor <email address hidden>

    Closes-Bug: #1838272
    Depends-On: https://review.opendev.org/c/openstack/puppet-tripleo/+/787771

    Change-Id: Ib1fd2fb89899b40b3ce2574af067006f566ef2ea

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/795148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795149

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 15.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 15.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/795148
Committed: https://opendev.org/openstack/puppet-tripleo/commit/0ee1aea3f57fbcefa7a33e0c5d19fea3262aab3f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0ee1aea3f57fbcefa7a33e0c5d19fea3262aab3f
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 23 16:52:45 2021 +0200

    Introduce nova virtlogd wrapper

    When nova_virtlogd container gets restarted the instance console auth files
    will not be reopened again by virtlogd. As a result either instances need
    to be restarted or live migrated to a different compute node to get new
    console logs messages logged again.
    Usually on receipt of SIGUSR1, virtlogd will re-exec() its binary, while
    maintaining all current logs and clients. This allows for live upgrades of
    the virtlogd service on non containerized environments where updates just
    by doing an RPM update.
    To reduce the likelihood in a containerized environment virtlogd should
    only be restarted on manual request, or on compute node reboot. It should
    not be restarted on a minor update without migration off instances.
    This introduces a nova_virtlogd_wrapper container and virtlogd wrapper
    script, to only restart virtlogd on either manual or compute node restart.

    Closes-Bug: #1838272

    Change-Id: I5192f8c306422d0966eef7544f719cc21172bd92
    Co-Authored-By: Rajesh Tailor <email address hidden>
    (cherry picked from commit f09aa48d95bc0fada01b1a011bc00e8c9b1e87b0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795149
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/0a690e519178043f3bb823126b79f7a03ac10a9a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0a690e519178043f3bb823126b79f7a03ac10a9a
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 23 17:08:30 2021 +0200

    Introduce nova virtlogd wrapper

    When nova_virtlogd container gets restarted the instance console auth
    files will not be reopened again by virtlogd. As a result either
    instances need to be restarted or live migrated to a different compute
    node to get new console logs messages logged again.
    Usually on receipt of SIGUSR1, virtlogd will re-exec() its binary,
    while maintaining all current logs and clients. This allows for live
    upgrades of the virtlogd service on non containerized environments
    where updates just by doing an RPM update.
    To reduce the likelihood in a containerized environment virtlogd
    should only be restarted on manual request, or on compute node reboot.
    It should not be restarted on a minor update without migration off
    instances.
    This introduces a nova_virtlogd_wrapper container and virtlogd wrapper
    script, to only restart virtlogd on either manual or compute node
    restart.

    With NovaEnableVirtlogdContainerWrapper the virtlogd wrapper can be
    disabled.

    Co-Authored-By: Rajesh Tailor <email address hidden>

    Closes-Bug: #1838272
    Depends-On: https://review.opendev.org/c/openstack/puppet-tripleo/+/795148

    Change-Id: Ib1fd2fb89899b40b3ce2574af067006f566ef2ea
    (cherry picked from commit 49415d04b2f97583d63004420b500d454f62386a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 16.0.0 release.