undercloud os-collect-config continually re-runs puppet

[Steven Hardy]

[root@undercloud ~]# systemctl status os-collect-config
● os-collect-config.service - Collect metadata and run hook commands.
   Loaded: loaded (/usr/lib/systemd/system/os-collect-config.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-03-15 06:45:20 EDT; 1h 40min ago
 Main PID: 681 (os-collect-conf)
    Tasks: 9
   CGroup: /system.slice/os-collect-config.service
           ├─ 681 /usr/bin/python /usr/bin/os-collect-config
           ├─3484 /usr/bin/python /usr/bin/os-refresh-config
           ├─3485 /bin/bash /usr/bin/dib-run-parts /usr/libexec/os-refresh-config/configure.d
           ├─3492 /bin/bash /usr/bin/dib-run-parts /usr/libexec/os-refresh-config/configure.d
           ├─3534 /bin/bash /usr/libexec/os-refresh-config/configure.d/50-puppet-stack-config
           ├─3535 /usr/bin/ruby /usr/bin/puppet apply --summarize --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp
           └─6470 /usr/bin/python2 /usr/bin/openstack token issue --format value

Mar 15 08:24:30 undercloud os-collect-config[681]: Notice: /Stage[main]/Nova::Conductor/Nova::Generic_service[conductor]/Service[nova-conductor]: Triggered... 1 events
Mar 15 08:24:47 undercloud os-collect-config[681]: Notice: /Stage[main]/Nova::Scheduler/Nova::Generic_service[scheduler]/Service[nova-scheduler]: Triggered... 1 events
Mar 15 08:24:49 undercloud os-collect-config[681]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: executed successfully
Mar 15 08:24:49 undercloud os-collect-config[681]: Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Triggered 'refresh' from 1 events
Mar 15 08:24:50 undercloud os-collect-config[681]: Notice: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Triggered 'refresh' from 1 events
Mar 15 08:24:50 undercloud os-collect-config[681]: Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Triggered 'refresh' from 2 events
Mar 15 08:25:11 undercloud os-collect-config[681]: Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 3 events
Mar 15 08:25:39 undercloud os-collect-config[681]: Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Triggered 'refresh' from 1 events
Mar 15 08:25:39 undercloud os-collect-config[681]: Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::end]: Triggered 'refresh' from 5 events
Mar 15 08:25:43 undercloud os-collect-config[681]: Notice: /Stage[main]/Glance::Deps/Anchor[glance::service::begin]: Triggered 'refresh' from 1 events

Every time o-c-c runs it re-applies puppet, I think the service should be disabled on the undercloud as there's nowhere for us to collect config from?

This breaks deployments because some services (such as mistral-engine) get restarted by puppet

[Steven Hardy (shardy)]

Ok this appears to be why it's re-running:

Mar 15 08:12:54 undercloud os-collect-config: /usr/libexec/os-refresh-config/post-configure.d/98-undercloud-setup: line 44: HOME: unbound variable
Mar 15 08:12:54 undercloud os-collect-config: [2018-03-15 08:12:54,798] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1]
Mar 15 08:12:54 undercloud os-collect-config: [2018-03-15 08:12:54,798] (os-refresh-config) [ERROR] Aborting...
Mar 15 08:12:54 undercloud os-collect-config: Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1

However the original undercloud install didn't report any errors in the undercloud_install.log:

2018-03-12 14:07:04 | 2018-03-12 14:07:04,979 INFO: [2018-03-12 14:07:04,974] (os-refresh-config) [INFO] Completed phase post-configure
2018-03-12 14:07:04 | 2018-03-12 14:07:04,980 INFO: os-refresh-config completed successfully

[Steven Hardy (shardy)]

This should be fixed by https://review.openstack.org/#/c/553395/ I think, I observed this on a stable/queens based build so proposed a backport at https://review.openstack.org/553395

Marking this as a duplicate of https://bugs.launchpad.net/tripleo/+bug/1754426