Brief Description
-----------------
After changing the OAM subnet IPs for the Controller, was unable to deploy platfirn-integ-app. While checking logs it was found that X509 certificate was invalid for the new subnet 172.28.239.244. see down logs. This made unable to download images from local registry.
Severity
--------
<Minor: OAM change then major>
Steps to Reproduce
------------------
Change the OAM IP
Expected Behavior
------------------
Should be able to download image from local registry without any problem and apply platofrm integ app.
Actual Behavior
----------------
Unable to apply application .
Reproducibility
---------------
yes Reproduciable
System Configuration
--------------------
AIO- Duplex: Stx 3.0
Timestamp/Logs
--------------
Attach the logs for debugging :
New OAm floating IP is 172.28.239.244.
Old OAM ip:172.28.235.244
sysinv 2020-06-15 20:49:09.730 162093 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds
sysinv 2020-06-15 20:49:09.775 162093 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244")
sysinv 2020-06-15 20:49:39.730 162093 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry
sysinv 2020-06-15 20:49:39.734 162093 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry
sysinv 2020-06-15 20:49:39.879 162093 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://172.28.239.244:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.244, not 172.28.239.244")
Workaround
----------
Manually update certificate as suggested by Austin.
backup your old key /cert
/etc/docker/certs.d/registry.local\:9001/registry-cert.crt
/etc/ssl/private/registry-cert.key
/etc/ssl/private/registry-cert.crt
1)change attached regisry-cent-extfile.cnf files accordingly .
IP.1 (mgr floating ip) and IP.2 (oam floating ip)
2) running command
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout /home/sysadmin/registry-cert.key -out /home/sysadmin/registry-cert.crt -config /home/sysadmin/regisry-cent-extfile.cnf
3)
copy registry-cert.key to /etc/ssl/private/registry-cert.key
copy registry-cert.crt to /etc/docker/certs.d/registry.local\:9001/registry-cert.crt and /etc/ssl/private/registry-cert.crt
openssl rsa -in registry-cert.key -out registry-cert-pkcs1.key
cp registry-cert-pkcs1.key /etc/ssl/private/
cp registry-cert.crt, registry-cert.key and registry-cert-pkcs1.key to /opt/platform/config/19.12/.
4) restart docker services and registry_token_server
For systemctl restart registry-token-server.service
5. Lock/Unlock controller
Closing as stx.3.0 is EOL as of Dec 2020