master: etc/pki/tls/certs/undercloud-192.168.24.2.pem]: Failed to generate additional resources using 'eval_generate': comparison of Array with Array failed

[wes hayutin]

It appears the ssl on the undercloud installs in master is failing.

018-01-27 00:38:21 | 2018-01-27 00:38:21,388 INFO: Notice: /Stage[main]/Ssh::Server::Config/Concat[/etc/ssh/sshd_config]/File[/etc/ssh/sshd_config]/content: content changed '{md5}d0e75c81e027e8b6b0631899c6c0e768' to '{md5}5822f16d9a2f8e603f9a16a083e4fcbe'
2018-01-27 00:38:21 | 2018-01-27 00:38:21,585 INFO: Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]: Triggered 'refresh' from 2 events
2018-01-27 00:38:21 | 2018-01-27 00:38:21,600 INFO: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-192.168.24.2.pem]/Concat_file[/etc/pki/tls/certs/undercloud-192.168.24.2.pem]: Failed to generate additional resources using 'eval_generate': comparison of Array with Array failed
2018-01-27 00:38:21 | 2018-01-27 00:38:21,605 INFO: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-192.168.24.2.pem]/File[/etc/pki/tls/certs/undercloud-192.168.24.2.pem]/ensure: created
2018-01-27 00:38:21 | 2018-01-27 00:38:21,693 INFO: Notice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Install[haproxy]/Package[haproxy]: Triggered 'refresh' from 1 events
2018-01-27 00:38:23 | 2018-01-27 00:38:23,060 INFO: Notice: /Stage[main]/Tripleo::Haproxy/Tripleo::Haproxy::Endpoint[keystone_admin]/Tripleo::Firewall::Rule[100 keystone_admin_haproxy]/Firewal

https://logs.rdoproject.org/07/472607/197/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z430801450f7342b7b72e4127c9dde14e/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-01-27_00_38_21

https://logs.rdoproject.org/07/472607/197/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Z430801450f7342b7b72e4127c9dde14e/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-01-27_00_38_46

Marking this with "alert" tag as the ovb check jobs are failing consistently.

[yatin (yatinkarel)]

Master periodic jobs are also affected.
Recently 2 commits merged in puppet-concat and they look suspicious:-
https://github.com/puppetlabs/puppetlabs-concat/commit/98d309c26daa4f71dcd397de8556ec0517631f16
https://github.com/puppetlabs/puppetlabs-concat/commit/7b33971ade0ad34946785b81a8fb4c5ecf471a64

[yatin (yatinkarel)]

https://github.com/puppetlabs/puppetlabs-concat/commit/98d309c26daa4f71dcd397de8556ec0517631f16 is causing the issue,
I reverted to the previous commit(d4857dfabe00ccbd158050e9245d074818a7cf15) and below worked:-
sudo puppet apply --modulepath /usr/share/openstack-puppet/modules/ -t -e 'include ::tripleo::certmonger::haproxy_dirs;ensure_resources('tripleo::certmonger::haproxy', hiera('tripleo::profile::base::haproxy::certificates_specs') , {'postsave_cmd' => "/usr/bin/instack-haproxy-cert-update '/etc/pki/tls/certs/undercloud-front.crt' '/etc/pki/tls/private/undercloud-front.key' /etc/pki/tls/certs/undercloud-192.168.24.2.pem undercloud-haproxy-public-cert"})'

[yatin (yatinkarel)]

Until the issue is fixed, proposed pin to good hash(d4857dfabe00ccbd158050e9245d074818a7cf15) in rdo: https://review.rdoproject.org/r/11621

[Alfredo Moralejo (amoralej)]

I'm not sure if the best solution is pin, fix in puppet modules or revert in puppet-concat. Could some puppeteer provide feedback?

[Emilien Macchi (emilienm)]

Ok for pin now, I'll look at the failure later in the day or week.