overcloud deploy failed due to Systemd start for pcsd failed

[chandan kumar]

On CentOS-8 fs035 and fs020 ovb multinode jobs are failing while doing overcloud deploy with following errors:

https://logserver.rdoproject.org/09/25909/1/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/bc847d4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz and

https://logserver.rdoproject.org/09/25909/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6777eaf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2020-03-16 08:37:28 | <13>Mar 16 08:37:25 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/User[hacluster]/groups: groups changed to ['haclient']
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Error: Systemd start for pcsd failed!
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: journalctl log for pcsd:
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: -- Logs begin at Mon 2020-03-16 08:26:33 UTC, end at Mon 2020-03-16 08:37:26 UTC. --
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:25 overcloud-controller-0 systemd[1]: Starting PCS GUI and remote configuration interface...
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: pcsd.service: Main process exited, code=exited, status=1/FAILURE
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: pcsd.service: Failed with result 'exit-code'.
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: Failed to start PCS GUI and remote configuration interface.
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Error: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: change from 'stopped' to 'running' failed: Systemd start for pcsd failed!
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: journalctl log for pcsd:
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: -- Logs begin at Mon 2020-03-16 08:26:33 UTC, end at Mon 2020-03-16 08:37:26 UTC. --
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:25 overcloud-controller-0 systemd[1]: Starting PCS GUI and remote configuration interface...
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: pcsd.service: Main process exited, code=exited, status=1/FAILURE
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: pcsd.service: Failed with result 'exit-code'.
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Mar 16 08:37:26 overcloud-controller-0 systemd[1]: Failed to start PCS GUI and remote configuration interface.
2020-03-16 08:37:28 | <13>Mar 16 08:37:26 puppet-user: Notice: /Stage[main]/Pacemaker::Service/Service[pcsd]: Triggered 'refresh' from 1 event

From jouralctl logs, I didnot find anything https://logserver.rdoproject.org/09/25909/1/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/bc847d4/logs/overcloud-controller-0/var/log/journal.txt.gz

Based on discussion with bandini on irc, it might be this:
<bandini> yeah usually it is a dns failure, lemme check
<bandini> mmh this is different: E, [2020-03-16T02:34:50.204 #00000] ERROR -- : Unable to start pcsd daemon, exiting: [Errno 2] No such file or directory: '/var/lib/pcsd/pcsd.crt'
<bandini> never seen that one

[Michele Baldessari (michele)]

Tested a downstream deploy and pcsd starts up just fine:
[root@controller-0 ~]# rpm -q pcs
pcs-0.10.2-4.el8.x86_64
[root@controller-0 ~]# rpm -q puppet-pacemaker puppet-tripleo
puppet-pacemaker-0.8.1-0.20200312111720.e1f66f2.el8ost.noarch
puppet-tripleo-11.4.1-0.20200312025814.116e8e8.el8ost.noarch

[Michele Baldessari (michele)]

Can we add the following hiera key 'pacemaker::corosync::pcsd_debug: true' to this promotion job?

[wes hayutin (weshayutin)]

moving to incomplete, can't repro

[Michele Baldessari (michele)]

So this seems to have resurfaced somehow. Yatin reproduced it with pcsd_debug set to true here:
https://logserver.rdoproject.org/19/713219/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/1b7bf6b/logs/

Interestingly enough pcsd debug true gave us zero additional information, so we'll need to do a pcs code walk through and understand what is going wrong here.

The normal pcsd start operation does create those files, so not sure what is going on here really.

[Rabi Mishra (rabi)]

Not sure if it's related. But I see

"2020-04-09 11:19:15 | Ignoring -days; not generating a certificate"

And is different from earlier passing jobs. May be the certs are getting corrupted with the change in https://review.opendev.org/#/c/717953/

https://logserver.rdoproject.org/53/717953/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ea401f/logs/undercloud/home/zuul/overcloud_create_ssl_cert.log.txt.gz

[Michele Baldessari (michele)]

So the root cause seems to be a bad image and is likely the same root cause for https://bugs.launchpad.net/tripleo/+bug/1871703

I was looking at an image from 1871703 in Wes' environment and I saw that /var/lib/pcsd is missing, so likely that is the root cause for this problem

[wes hayutin (weshayutin)]

Chandan could not repro

[Rabi Mishra (rabi)]

> Chandan could not repro

@wes, What does this mean? Are you saying there is no issue with the image as mentioned in comment #6?

We've noticed these failures today with the ovb jobs.

[wes hayutin (weshayutin)]

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/5c2765d/logs/overcloud-controller-0/var/log/extra/failed_services.txt.gz

[Michele Baldessari (michele)]

So the error is a bit different this time but it is still an image problem:
Apr 13 11:53:15 overcloud-controller-0 pcsd[35068]: FileNotFoundError: [Errno 2] No such file or directory: '/var/log/pcsd/pcsd.log'

So to me this implies that /var/log/pcsd has been removed, since it normally is part of the pcs rpm:
[root@controller-0 ~]# rpm -ql pcs |grep "var/log"
/var/log/pcsd

[Rafael Folco (rafaelfolco)]

seen this on https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/7f5fe14/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: Error: Systemd start for pcsd failed!
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: journalctl log for pcsd:
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: -- Logs begin at Sun 2020-05-31 18:53:38 UTC, end at Sun 2020-05-31 19:08:47 UTC. --
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:46 overcloud-controller-0 systemd[1]: Starting PCS GUI and remote configuration interface...
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: pcsd.service: Main process exited, code=exited, status=1/FAILURE
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: pcsd.service: Failed with result 'exit-code'.
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: Failed to start PCS GUI and remote configuration interface.
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: Error: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: change from 'stopped' to 'running' failed: Systemd start for pcsd failed!
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: journalctl log for pcsd:
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: -- Logs begin at Sun 2020-05-31 18:53:38 UTC, end at Sun 2020-05-31 19:08:47 UTC. --
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:46 overcloud-controller-0 systemd[1]: Starting PCS GUI and remote configuration interface...
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: pcsd.service: Main process exited, code=exited, status=1/FAILURE
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: pcsd.service: Failed with result 'exit-code'.
2020-05-31 19:08:50 | <13>May 31 19:08:47 puppet-user: May 31 19:08:47 overcloud-controller-0 systemd[1]: Failed to start PCS GUI and remote configuration interface.

[wes hayutin (weshayutin)]

This is a CI issue or DIB..

[wes hayutin (weshayutin)]

Same issue as https://bugs.launchpad.net/tripleo/+bug/1879766

missing files..

Fix the image build and check please :)