Comment 2 for bug 1673235

It seems like *.pem files are created before mysqld can even check if auto_generate_certs and sha256_password_auto_generate_rsa_keys are OFF.

Started mysqld with no *.pem files in the datadir.
root@test1:/var/lib/mysql# ls
auto.cnf ib_buffer_pool ibdata1 ib_logfile0 ib_logfile1 mysql performance_schema sys

Error log shows this upon startup:
< snipped >
2017-03-16T04:26:34.204904Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2017-03-16T04:26:34.204978Z 0 [Note] Skipping generation of SSL certificates as --auto_generate_certs is set to OFF.
2017-03-16T04:26:34.216585Z 0 [Warning] CA certificate ca.pem is self signed.
2017-03-16T04:26:34.216691Z 0 [Note] Skipping generation of RSA key pair as --sha256_password_auto_generate_rsa_keys is set to OFF.
< snipped >

Ended up with new *.pem files in the datadir.
root@test1:/var/lib/mysql# ls
auto.cnf client-cert.pem ibdata1 ibtmp1 performance_schema server-cert.pem xb_doublewrite
ca-key.pem client-key.pem ib_logfile0 mysql private_key.pem server-key.pem
ca.pem ib_buffer_pool ib_logfile1 mysqld_safe.pid public_key.pem sys