Minor update can result in instances get rebooted with NovaResumeGuestsStateOnHostBoot: True

[Martin Schuppert]

* Ansible host prepare step which started libvirt-guests correct:
Oct 9 11:40:41 overcloud-compute-15 ansible-stat: Invoked with checksum_algorithm=sha1 get_checksum=True follow=False checksum_algo=sha1 path=/etc/systemd/system/libvirt-guests.service get_md5=None get_mime=True get_attributes=True
Oct 9 11:40:41 overcloud-compute-15 ansible-copy: Invoked with directory_mode=None force=True remote_src=None _original_basename=tmpcf2YY1 owner=None follow=False local_follow=None group=None unsafe_writes=None setype=None content=NOT_LOGGING_PARAMETER serole=None dest=/etc/
systemd/system/libvirt-guests.service selevel=None regexp=None validate=None src=/root/.ansible/tmp/ansible-tmp-1570614041.02-199123311360310/source checksum=b05237d34e522f44407f65882217e7e518b356dc seuser=None delimiter=None mode=None attributes=None backup=False
Oct 9 11:40:41 overcloud-compute-15 ansible-systemd: Invoked with no_block=False force=None name=libvirt-guests enabled=True daemon_reload=True state=started masked=None user=False
Oct 9 11:40:41 overcloud-compute-15 systemd: Reloading.
Oct 9 11:40:41 overcloud-compute-15 systemd: Started Flexible Branding Service.
Oct 9 11:40:42 overcloud-compute-15 systemd: Reloading.
Oct 9 11:40:42 overcloud-compute-15 systemd: Reached target Libvirt guests shutdown.
Oct 9 11:40:42 overcloud-compute-15 systemd: Starting Suspend/Resume Running libvirt Guests...
Oct 9 11:40:42 overcloud-compute-15 systemd: Started Flexible Branding Service.
Oct 9 11:40:42 overcloud-compute-15 systemd: Started Suspend/Resume Running libvirt Guests.
...
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: TASK [is Nova Resume Guests State On Host Boot enabled] ************************
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: ok: [localhost]
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: TASK [libvirt-guests unit to stop nova_compute container before shutdown VMs] ***
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: changed: [localhost]
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: TASK [libvirt-guests enable VM shutdown on compute reboot/shutdown] ************
Oct 9 11:40:58 overcloud-compute-15 os-collect-config: changed: [localhost]

* Then puppet-tripleo got called for the added OS::TripleO::Services::NovaLibvirtGuests service to the compute role:
Oct 9 11:53:56 overcloud-compute-15 puppet-user[560442]: Compiled catalog for overcloud-compute-15.xyz in environment production in 2.96 seconds
Oct 9 11:53:57 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_Compute]/ensure) created
Oct 9 11:53:57 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Tripleo::Profile::Base::Nova::Compute::Libvirt_guests/File[/etc/systemd/system/virt-guest-shutdown.target.wants]/ensure) created
Oct 9 11:53:58 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns) executed successfully
Oct 9 11:53:58 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Tripleo::Profile::Base::Nova::Compute::Libvirt_guests/Systemd::Unit_file[paunch-container-shutdown.service]/File[/etc/systemd/system/virt-guest-shutdown.target.wants/paunch-container-shutdown.service]/ensure) created

- Note we also do a systemctl daemon reload in [1]
Oct 9 11:53:58 overcloud-compute-15 systemd: Reloading.
Oct 9 11:53:59 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Systemd::Systemctl::Daemon_reload/Exec[systemctl-daemon-reload]) Triggered 'refresh' from 1 events
Oct 9 11:53:59 overcloud-compute-15 systemd: Started Flexible Branding Service.
Oct 9 11:53:59 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]/ensure) created
Oct 9 11:53:59 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]/ensure) created
Oct 9 11:53:59 overcloud-compute-15 puppet-user[560442]: (/Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]/ensure) created

- the libvirt-guests stop was a result of the puppet-tripleo/puppet-nova run:
Oct 9 11:53:59 overcloud-compute-15 systemd: Stopping Suspend/Resume Running libvirt Guests...
Oct 9 11:54:05 overcloud-compute-15 journal: 2019-10-09 09:54:05.820+0000: 557266: info : libvirt version: 4.5.0, package: 23.el7_7.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-08-16-11:33:27, x86-vm-28.build.eng.bos.redhat.com)
Oct 9 11:54:05 overcloud-compute-15 journal: 2019-10-09 09:54:05.820+0000: 557266: info : hostname: overcloud-compute-15
Oct 9 11:54:05 overcloud-compute-15 journal: 2019-10-09 09:54:05.820+0000: 557266: error : virNetSocketReadWire:1806 : End of file while reading data: Input/output error
Oct 9 11:54:05 overcloud-compute-15 dockerd-current: time="2019-10-09T11:54:05.931711967+02:00" level=warning msg="dcb3e6206fe2a41e7d9888fe9a8fe0577516e50ac5f6fb18d6cb8494d0d3e26b cleanup: failed to unmount secrets: invalid argument"
Oct 9 11:54:05 overcloud-compute-15 docker: nova_compute
Oct 9 11:54:06 overcloud-compute-15 libvirt-guests.sh: Running guests on default URI: instance-0000111d, instance-00000e59, instance-000015ea, instance-00000d36
Oct 9 11:54:06 overcloud-compute-15 libvirt-guests.sh: Shutting down guests on default URI...
Oct 9 11:54:06 overcloud-compute-15 libvirt-guests.sh: Starting shutdown on guest: instance-0000111d
Oct 9 11:54:08 overcloud-compute-15 libvirt-guests.sh: Waiting for guest instance-0000111d to shut down, 300 seconds left
...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/690055

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/690631

[OpenStack Infra (hudson-openstack)]

Change abandoned by Martin Schuppert (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/690631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690055
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=9d3af781e74a6eb537ad53ca504b6a5994af6b1d
Submitter: Zuul
Branch: master

commit 9d3af781e74a6eb537ad53ca504b6a5994af6b1d
Author: Martin Schuppert <email address hidden>
Date: Tue Oct 22 13:50:03 2019 +0200

    Only configure libvirt-guests if enabled

    If NovaResumeGuestsStateOnHostBoot is not true, there is no
    reason to configure libvirt-guests.

    Also a customized libvirt-guests systemd unit file is added
    via [1] which already has the dependencies configured. Therefore
    there is no need to manage them via puppet again.

    Related-Bug: #1849264
    Depends-On: https://review.opendev.org/690016

    [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-compute-container-puppet.yaml#L777

    Change-Id: Ic127937ad83eebb47e34c5f36c826f8dcf1048a9

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/691179

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/691247

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/691248

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691179
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=5a9ea4011115010f00b3a898c135624f6efb4478
Submitter: Zuul
Branch: stable/train

commit 5a9ea4011115010f00b3a898c135624f6efb4478
Author: Martin Schuppert <email address hidden>
Date: Tue Oct 22 13:50:03 2019 +0200

    Only configure libvirt-guests if enabled

    If NovaResumeGuestsStateOnHostBoot is not true, there is no
    reason to configure libvirt-guests.

    Also a customized libvirt-guests systemd unit file is added
    via [1] which already has the dependencies configured. Therefore
    there is no need to manage them via puppet again.

    Related-Bug: #1849264
    Depends-On: https://review.opendev.org/690621

    [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-compute-container-puppet.yaml#L777

    Change-Id: Ic127937ad83eebb47e34c5f36c826f8dcf1048a9
    (cherry picked from commit 9d3af781e74a6eb537ad53ca504b6a5994af6b1d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690631
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=71664348d4bd7c4e649df9aaba0a48a19e392e49
Submitter: Zuul
Branch: stable/stein

commit 71664348d4bd7c4e649df9aaba0a48a19e392e49
Author: Martin Schuppert <email address hidden>
Date: Tue Oct 22 13:50:03 2019 +0200

    Only configure libvirt-guests if enabled

    If NovaResumeGuestsStateOnHostBoot is not true, there is no
    reason to configure libvirt-guests.

    Also a customized libvirt-guests systemd unit file is added
    via [1] which already has the dependencies configured. Therefore
    there is no need to manage them via puppet again.

    Related-Bug: #1849264
    Depends-On: https://review.opendev.org/690624

    [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-compute-container-puppet.yaml#L777

    Change-Id: Ic127937ad83eebb47e34c5f36c826f8dcf1048a9
    (cherry picked from commit 9d3af781e74a6eb537ad53ca504b6a5994af6b1d)
    (cherry picked from commit 5a9ea4011115010f00b3a898c135624f6efb4478)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691247
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0d7b7bfc4696436a83a22693f43b767183449ede
Submitter: Zuul
Branch: stable/rocky

commit 0d7b7bfc4696436a83a22693f43b767183449ede
Author: Martin Schuppert <email address hidden>
Date: Tue Oct 22 13:50:03 2019 +0200

    Only configure libvirt-guests if enabled

    If NovaResumeGuestsStateOnHostBoot is not true, there is no
    reason to configure libvirt-guests.

    Also a customized libvirt-guests systemd unit file is added
    via [1] which already has the dependencies configured. Therefore
    there is no need to manage them via puppet again.

    Related-Bug: #1849264
    Depends-On: https://review.opendev.org/690625

    [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-compute-container-puppet.yaml#L777

    Change-Id: Ic127937ad83eebb47e34c5f36c826f8dcf1048a9
    (cherry picked from commit 9d3af781e74a6eb537ad53ca504b6a5994af6b1d)
    (cherry picked from commit 5a9ea4011115010f00b3a898c135624f6efb4478)
    (cherry picked from commit 71664348d4bd7c4e649df9aaba0a48a19e392e49)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691248
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=00fcfd49f0df60f58d1021f0e046790412023dbb
Submitter: Zuul
Branch: stable/queens

commit 00fcfd49f0df60f58d1021f0e046790412023dbb
Author: Martin Schuppert <email address hidden>
Date: Tue Oct 22 13:50:03 2019 +0200

    Only configure libvirt-guests if enabled

    If NovaResumeGuestsStateOnHostBoot is not true, there is no
    reason to configure libvirt-guests.

    Also a customized libvirt-guests systemd unit file is added
    via [1] which already has the dependencies configured. Therefore
    there is no need to manage them via puppet again.

    Related-Bug: #1849264
    Depends-On: https://review.opendev.org/690626

    [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-compute-container-puppet.yaml#L777

    Change-Id: Ic127937ad83eebb47e34c5f36c826f8dcf1048a9
    (cherry picked from commit 9d3af781e74a6eb537ad53ca504b6a5994af6b1d)
    (cherry picked from commit 5a9ea4011115010f00b3a898c135624f6efb4478)
    (cherry picked from commit 71664348d4bd7c4e649df9aaba0a48a19e392e49)
    (cherry picked from commit 0d7b7bfc4696436a83a22693f43b767183449ede)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova stein-eol release.