AIO-Duplex-Extended: After changing OAM subnet get x509: certificate is invalid for new subnet

[Yatindra Shashi]

Brief Description
-----------------
After changing the OAM subnet IPs for the Controller, was unable to deploy platfirn-integ-app. While checking logs it was found that X509 certificate was invalid for the new subnet 172.28.239.244. see down logs. This made unable to download images from local registry.

Severity
--------

<Minor: OAM change then major>

Steps to Reproduce
------------------
Change the OAM IP

Expected Behavior
------------------
Should be able to download image from local registry without any problem and apply platofrm integ app.

Actual Behavior
----------------
Unable to apply application .

Reproducibility
---------------
yes Reproduciable

System Configuration
--------------------
AIO- Duplex: Stx 3.0

Timestamp/Logs
--------------
Attach the logs for debugging :
New OAm floating IP is 172.28.239.244.
Old OAM ip:172.28.235.244

sysinv 2020-06-15 20:49:09.730 162093 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds
sysinv 2020-06-15 20:49:09.775 162093 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244")
sysinv 2020-06-15 20:49:39.730 162093 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry
sysinv 2020-06-15 20:49:39.734 162093 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry
sysinv 2020-06-15 20:49:39.879 162093 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244")

 Workaround
 ----------
Manually update certificate as suggested by Austin.

 backup your old key /cert
/etc/docker/certs.d/registry.local\:9001/registry-cert.crt
/etc/ssl/private/registry-cert.key
/etc/ssl/private/registry-cert.crt

1)change attached regisry-cent-extfile.cnf files accordingly .
   IP.1 (mgr floating ip) and IP.2 (oam floating ip)

2) running command
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout /home/sysadmin/registry-cert.key -out /home/sysadmin/registry-cert.crt -config /home/sysadmin/regisry-cent-extfile.cnf

3)

copy registry-cert.key to /etc/ssl/private/registry-cert.key
copy registry-cert.crt to /etc/docker/certs.d/registry.local\:9001/registry-cert.crt and /etc/ssl/private/registry-cert.crt
openssl rsa -in registry-cert.key -out registry-cert-pkcs1.key
 cp registry-cert-pkcs1.key /etc/ssl/private/
 cp registry-cert.crt, registry-cert.key and registry-cert-pkcs1.key to /opt/platform/config/19.12/.

4) restart docker services and registry_token_server
   For systemctl restart registry-token-server.service

5. Lock/Unlock controller

[Ghada Khalil (gkhalil)]

Closing as stx.3.0 is EOL as of Dec 2020