Puppet Agent Master sync issue

[Shweta P]

Puppet agents on the controller or the compute nodes have the following issue while communicating to the pupet master on the build node.

SYSLOG output

(/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: The certificate retrieved from the master does not match the agent's private key.#012Certificate fingerprint: 04:F2:9F:D8:C9:CA:8B:C9:F7:F9:0E:1F:34:28:58:C4#012To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.#012On the master:#012 puppet cert clean p4-control01.ci.ctocllab.cisco.com#012On the agent:#012 rm -f /var/lib/puppet/ssl/certs/p4-control01.ci.ctocllab.cisco.com.pem#012 puppet agent -t
Dec 10 15:20:30 p4-control01 puppet-agent[1170]: (/File[/var/lib/puppet/lib]) Could not evaluate: SSL_CTX_use_PrivateKey:: key values mismatch Could not retrieve file metadata for puppet://pod4node1.ci.ctocllab.cisco.com/plugins: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Using cached catalog
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not retrieve catalog; skipping run
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Creating a new SSL certificate request for p4-control01.ci.ctocllab.cisco.com
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Certificate Request fingerprint (md5): 6C:D0:C0:F0:80:6B:1A:9A:88:64:86:23:B6:77:1E:29
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Caching certificate for p4-control01.ci.ctocllab.cisco.com
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Retrieving plugin
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: (/File[/var/lib/puppet/lib]) Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client Could not retrieve file metadata for puppet://pod4node1.ci.ctocllab.cisco.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:23 p4-control01 puppet-agent[1734]: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:23 p4-control01 puppet-agent[1734]: Not using cache on failed catalo

Workaround
On the controller and the compute nodes
stop the puppet agent
rm -f /var/lib/puppet/ssl/
Its not sufficient to just delete the specific certificate(*.pem file ), the entire ssl directory needs to be deleted.
start the puppet agent.

Related bug
http://projects.puppetlabs.com/issues/15290

[kylefoo (fung933)]

I have encountered one of the issues you had.

(/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client

After removing the ssl directory and regenerate the cert, we have passed the key matching issue. But above error occurred due to the master and agent's ntp setting that is not being synchronized. just do the following on your agent:

ntpd -q [IP address of your master]
ntpdate [IP address of your master]

it is provided that your master has the right ntp settings. then, do

puppetd --waitforce 60 --test --server [IP address or hostname of your master]