Rerunning the overcloud deploy command with no changes restarts a truckload of containers (first seen this via https://bugzilla.redhat.com/show_bug.cgi?id=1612960).
So we really have three separate issues here. Below is the list of all the containers
that may restart needlessly (at least what I have observed in my tests):
A) cron category:
ceilometer_agent_notification
cinder_api
cinder_api_cron
cinder_scheduler
heat_api
heat_api_cfn
heat_api_cron
heat_engine
keystone
keystone_cron
logrotate_crond
nova_api
nova_api_cron
nova_conductor
nova_consoleauth
nova_metadata
nova_scheduler
nova_vnc_proxy
openstack-cinder-volume-docker-0
panko_api
These end up being restarted because in the config volume for the container there is
a cron file and cron files are generated with a timestamp inside:
$ cat /var/lib/config-data/puppet-generated/keystone/var/spool/cron/keystone
# HEADER: This file was autogenerated at 2018-08-07 11:44:57 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: keystone-manage token_flush
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 * * * * keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1
The timestamp is unfortunately hard coded into puppet in both the cron provider and the parsedfile
provider:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/cron/crontab.rb#L127
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/parsedfile.rb#L104
One possible fix would be to change the tar command in docker-puppet.py to be something like
this:
tar -c -f - /var/lib/config-data/${NAME} --mtime='1970-01-01' | tar xO | sed '/#.*HEADER.*/d'
Note that the 'tar xO' is needed in order to force text output and not use the
native tar format (if you don't the sed won't work).
B) swift category:
swift_account_auditor
swift_account_reaper
swift_account_replicator
swift_account_server
swift_container_auditor
swift_container_replicator
swift_container_server
swift_container_updater
swift_object_auditor
swift_object_expirer
swift_object_replicator
swift_object_server
swift_object_updater
swift_proxy
swift_rsync
So the swift containers restart because when recalculating the md5 over the
/var/lib/config-data/puppet-generated/swift folder we also include:
B.1) /etc/swift/backups/... which is a folder which over time collects backup of the ringfiles
B.2) /etc/swift/*.gz it seems that the *.gz files seem to change over time
I see two potential fixes here:
B.3) We simply exclude rings and their backups from the tar commands:
EXCL="--exclude='*/etc/swift/backups/*' --exclude='*/etc/swift/*.gz'"
tar -c -f - /var/lib/config-data/${NAME} --mtime='1970-01-01' $EXCL | tar xO | sed '/^#.*HEADER.*/d'
B.4) We stop storing the rings under /etc/swift and we store them in some other place ?
I lack the necessary swift knowledge to add anything too useful here.
C) libvirt category:
nova_compute
nova_libvirt
nova_migration_target
nova_virtlogd
This one seems to be due to the fact that the /etc/libvirt/passwd.db file contains a timestamp or somehow
is just different after a redeploy:
[root@compute-1 nova_libvirt]# git diff cb2441bb1caf7572ccfd870561dcc29d7819ba04..0c7441f30926b111603ce4d4b60c6000fe49d290 .
diff --git a/puppet-generated/nova_libvirt/etc/libvirt/passwd.db b/puppet-generated/nova_libvirt/etc/libvirt/passwd.db
index 66e2024..6ffa492 100644
Binary files a/puppet-generated/nova_libvirt/etc/libvirt/passwd.db and b/puppet-generated/nova_libvirt/etc/libvirt/passwd.db differ
(I have not exactly understood why yet because in my quick test without tls a simple 'saslpasswd2 -d -a libvirt -u overcloud migration' did not change the md5 for me, but maybe I am missing something)
Excluding this file from the tar command is probably a bad idea because if you do indeed change the password of the migration user then you probably do want the libvirtd container to restart (actually you likely want only that one to restart, as opposed to all nova* containers on the compute which is what happens now)
Just to make sure that the above theories are all correct I used the following tar commands:
EXCL="--exclude='*/etc/swift/backups/*' --exclude='*/etc/swift/*.gz' --exclude '*/etc/libvirt/passwd.db'"
tar -c -f - /var/lib/config-data/${NAME} --mtime='1970-01-01' $EXCL | tar xO | sed '/^#.*HEADER.*/d' | md5sum | awk '{print $1}' > /var/lib/config-data/${NAME}.md5sum
tar -c -f - /var/lib/config-data/puppet-generated/${NAME} --mtime='1970-01-01' $EXCL | tar xO | sed '/^#.*HEADER.*/d' | md5sum | awk '{print $1}' > /var/lib/config-data/puppet-generated/${NAME}.md5sum
And I observed no spurious restarts.
Related fix proposed to branch: master
Review: https:/