Activity log for bug #2003007

Date Who What changed Old value New value Message
2023-01-16 18:19:54 gulsum atici bug added bug
2023-01-16 21:20:39 gulsum atici description Hello, Juju 2.9.37 is installed in a VM and charms were running in the osm model which has juju version 2.9.34. installed: 2.9.37 (21315) 96MB classic ubuntu@osm12:/usr/share/osm-devops/installers$ juju status Model Controller Cloud/Region Version SLA Timestamp osm osm-vca microk8s/localhost 2.9.34 unsupported 17:13:21Z App Version Status Scale Charm Channel Rev Address Exposed Message grafana res:image@9b5a5a8 active 1 osm-grafana 12.0/stable 100 10.152.183.234 no ready kafka active 1 kafka-k8s latest/stable 5 10.152.183.30 no keystone active 1 osm-keystone latest/stable 5 10.152.183.188 no lcm opensourcemano/lcm:12 active 1 osm-lcm 12.0/stable 113 10.152.183.166 no ready mariadb mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s stable 35 10.152.183.251 no ready mon opensourcemano/mon:12 active 1 osm-mon 12.0/stable 106 10.152.183.186 no ready mongodb library/mongo@11b4907 active 1 mongodb-k8s latest/stable 14 10.152.183.128 no nbi opensourcemano/nbi:12 active 1 osm-nbi 12.0/stable 1 10.152.183.240 no ready ng-ui opensourcemano/ng-ui:12 active 1 osm-ng-ui 12.0/stable 1 10.152.183.65 no ready pla opensourcemano/pla:12 active 1 osm-pla latest/stable 1 10.152.183.141 no ready pol opensourcemano/pol:12 active 1 osm-pol 12.0/stable 1 10.152.183.215 no ready prometheus res:backup-image@7996084 active 1 osm-prometheus 12.0/stable 1 10.152.183.33 no ready ro opensourcemano/ro:12 active 1 osm-ro 12.0/stable 4 10.152.183.178 no ready zookeeper active 1 zookeeper-k8s latest/stable 10 10.152.183.34 no Unit Workload Agent Address Ports Message grafana/0* active idle 10.1.47.108 3000/TCP ready kafka/0* active idle 10.1.47.104 keystone/0* active idle 10.1.47.124 lcm/0* active idle 10.1.47.100 9999/TCP ready mariadb/0* active idle 10.1.47.106 3306/TCP ready mon/1* active idle 10.1.47.93 8000/TCP ready mongodb/0* active idle 10.1.47.101 27017/TCP nbi/0* active idle 10.1.47.121 9999/TCP ready ng-ui/0* active idle 10.1.47.120 80/TCP ready pla/0* active idle 10.1.47.103 9999/TCP ready pol/0* active idle 10.1.47.126 9999/TCP ready prometheus/0* active idle 10.1.47.112 9090/TCP ready ro/0* active idle 10.1.47.91 9090/TCP ready zookeeper/0* active idle 10.1.47.119 Controller has been upgraded by running: juju upgrade-controller --agent-version 2.0.37 Then model has been upgraded with command: juju upgrade-model --agent-version 2.0.37 After upgrade, some of the units are stuck at error state with the following logs: https://pastebin.ubuntu.com/p/WYBFcRcdKr/ ubuntu@osm12:/usr/share/osm-devops/installers$ juju status Model Controller Cloud/Region Version SLA Timestamp osm osm-vca microk8s/localhost 2.9.37 unsupported 17:27:02Z App Version Status Scale Charm Channel Rev Address Exposed Message grafana res:image@9b5a5a8 active 1 osm-grafana 12.0/stable 100 10.152.183.234 no ready kafka waiting 0/1 kafka-k8s latest/stable 5 10.152.183.30 no installing agent keystone waiting 0/1 osm-keystone latest/stable 5 10.152.183.188 no installing agent lcm opensourcemano/lcm:12 waiting 1 osm-lcm 12.0/stable 113 10.152.183.166 no mariadb mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s stable 35 10.152.183.251 no ready mon opensourcemano/mon:12 active 1 osm-mon 12.0/stable 106 10.152.183.186 no ready mongodb library/mongo@11b4907 active 1 mongodb-k8s latest/stable 14 10.152.183.128 no nbi opensourcemano/nbi:12 waiting 2/1 osm-nbi 12.0/stable 1 10.152.183.240 no ng-ui opensourcemano/ng-ui:12 active 1 osm-ng-ui 12.0/stable 1 10.152.183.65 no ready pla opensourcemano/pla:12 active 1 osm-pla latest/stable 1 10.152.183.141 no ready pol opensourcemano/pol:12 waiting 1 osm-pol 12.0/stable 1 10.152.183.215 no prometheus res:backup-image@7996084 active 1 osm-prometheus 12.0/stable 1 10.152.183.33 no ready ro opensourcemano/ro:12 active 1 osm-ro 12.0/stable 4 10.152.183.178 no ready zookeeper waiting 0/1 zookeeper-k8s latest/stable 10 10.152.183.34 no installing agent Unit Workload Agent Address Ports Message grafana/1* active idle 10.1.47.86 3000/TCP ready kafka/0 error lost 10.1.47.83 crash loop backoff: back-off 5m0s restarting failed container=charm-init pod=kafka-0_osm(2b65e1aa-2878-49d6-9730-396d65cc6631) keystone/0 error lost 10.1.47.122 crash loop backoff: back-off 5m0s restarting failed container=charm-init pod=keystone-0_osm(23991780-fd5c-43ee-9141-3f772eb85e54) lcm/1* error idle 10.1.47.96 9999/TCP crash loop backoff: back-off 5m0s restarting failed container=lcm pod=lcm-559f79cfbb-blcsj_osm(1bc7397b-28b7-448a-b216-901f52336f78) mariadb/0* active idle 10.1.47.84 3306/TCP ready mon/2* active idle 10.1.47.116 8000/TCP ready mongodb/0* active idle 10.1.47.82 27017/TCP nbi/0* active idle 10.1.47.121 9999/TCP ready nbi/1 waiting idle 10.1.47.92 9999/TCP waiting for container ng-ui/1* active idle 10.1.47.90 80/TCP ready pla/1* active idle 10.1.47.125 9999/TCP ready pol/1* error idle 10.1.47.77 9999/TCP crash loop backoff: back-off 5m0s restarting failed container=pol pod=pol-799dbfc869-qjv9h_osm(8b048123-9354-4cda-b4c0-6875ee90aa48) prometheus/0* active idle 10.1.47.123 9090/TCP ready ro/1* active idle 10.1.47.78 9090/TCP ready zookeeper/0 error lost 10.1.47.95 container error: Could you help for this problem ? How to reproduce the problem: 1. Install OSM: wget https://osm-download.etsi.org/ftp/osm-13.0-thirteen/install_osm.sh chmod +x install_osm.sh ./install_osm.sh --charmed 2. Upgrade controller juju upgrade-controller --agent-version 2.0.37 3. Upgrade model juju upgrade-model --agent-version 2.0.37 4. Check the status of applications juju status, juju debug-log Many Thanks! Gulsum Hello, Juju 2.9.37 is installed in a VM and charms were running in the osm model which has juju version 2.9.34. installed: 2.9.37 (21315) 96MB classic ubuntu@osm12:/usr/share/osm-devops/installers$ juju status Model Controller Cloud/Region Version SLA Timestamp osm osm-vca microk8s/localhost 2.9.34 unsupported 17:13:21Z App Version Status Scale Charm Channel Rev Address Exposed Message grafana res:image@9b5a5a8 active 1 osm-grafana 12.0/stable 100 10.152.183.234 no ready kafka active 1 kafka-k8s latest/stable 5 10.152.183.30 no keystone active 1 osm-keystone latest/stable 5 10.152.183.188 no lcm opensourcemano/lcm:12 active 1 osm-lcm 12.0/stable 113 10.152.183.166 no ready mariadb mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s stable 35 10.152.183.251 no ready mon opensourcemano/mon:12 active 1 osm-mon 12.0/stable 106 10.152.183.186 no ready mongodb library/mongo@11b4907 active 1 mongodb-k8s latest/stable 14 10.152.183.128 no nbi opensourcemano/nbi:12 active 1 osm-nbi 12.0/stable 1 10.152.183.240 no ready ng-ui opensourcemano/ng-ui:12 active 1 osm-ng-ui 12.0/stable 1 10.152.183.65 no ready pla opensourcemano/pla:12 active 1 osm-pla latest/stable 1 10.152.183.141 no ready pol opensourcemano/pol:12 active 1 osm-pol 12.0/stable 1 10.152.183.215 no ready prometheus res:backup-image@7996084 active 1 osm-prometheus 12.0/stable 1 10.152.183.33 no ready ro opensourcemano/ro:12 active 1 osm-ro 12.0/stable 4 10.152.183.178 no ready zookeeper active 1 zookeeper-k8s latest/stable 10 10.152.183.34 no Unit Workload Agent Address Ports Message grafana/0* active idle 10.1.47.108 3000/TCP ready kafka/0* active idle 10.1.47.104 keystone/0* active idle 10.1.47.124 lcm/0* active idle 10.1.47.100 9999/TCP ready mariadb/0* active idle 10.1.47.106 3306/TCP ready mon/1* active idle 10.1.47.93 8000/TCP ready mongodb/0* active idle 10.1.47.101 27017/TCP nbi/0* active idle 10.1.47.121 9999/TCP ready ng-ui/0* active idle 10.1.47.120 80/TCP ready pla/0* active idle 10.1.47.103 9999/TCP ready pol/0* active idle 10.1.47.126 9999/TCP ready prometheus/0* active idle 10.1.47.112 9090/TCP ready ro/0* active idle 10.1.47.91 9090/TCP ready zookeeper/0* active idle 10.1.47.119 Controller has been upgraded by running: juju upgrade-controller --agent-version 2.0.37 Then model has been upgraded with command: juju upgrade-model --agent-version 2.0.37 After upgrade, some of the units are stuck at error state with the following logs: https://pastebin.ubuntu.com/p/WYBFcRcdKr/ ubuntu@osm12:/usr/share/osm-devops/installers$ juju status Model Controller Cloud/Region Version SLA Timestamp osm osm-vca microk8s/localhost 2.9.37 unsupported 17:27:02Z App Version Status Scale Charm Channel Rev Address Exposed Message grafana res:image@9b5a5a8 active 1 osm-grafana 12.0/stable 100 10.152.183.234 no ready kafka waiting 0/1 kafka-k8s latest/stable 5 10.152.183.30 no installing agent keystone waiting 0/1 osm-keystone latest/stable 5 10.152.183.188 no installing agent lcm opensourcemano/lcm:12 waiting 1 osm-lcm 12.0/stable 113 10.152.183.166 no mariadb mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s stable 35 10.152.183.251 no ready mon opensourcemano/mon:12 active 1 osm-mon 12.0/stable 106 10.152.183.186 no ready mongodb library/mongo@11b4907 active 1 mongodb-k8s latest/stable 14 10.152.183.128 no nbi opensourcemano/nbi:12 waiting 2/1 osm-nbi 12.0/stable 1 10.152.183.240 no ng-ui opensourcemano/ng-ui:12 active 1 osm-ng-ui 12.0/stable 1 10.152.183.65 no ready pla opensourcemano/pla:12 active 1 osm-pla latest/stable 1 10.152.183.141 no ready pol opensourcemano/pol:12 waiting 1 osm-pol 12.0/stable 1 10.152.183.215 no prometheus res:backup-image@7996084 active 1 osm-prometheus 12.0/stable 1 10.152.183.33 no ready ro opensourcemano/ro:12 active 1 osm-ro 12.0/stable 4 10.152.183.178 no ready zookeeper waiting 0/1 zookeeper-k8s latest/stable 10 10.152.183.34 no installing agent Unit Workload Agent Address Ports Message grafana/1* active idle 10.1.47.86 3000/TCP ready kafka/0 error lost 10.1.47.83 crash loop backoff: back-off 5m0s restarting failed container=charm-init pod=kafka-0_osm(2b65e1aa-2878-49d6-9730-396d65cc6631) keystone/0 error lost 10.1.47.122 crash loop backoff: back-off 5m0s restarting failed container=charm-init pod=keystone-0_osm(23991780-fd5c-43ee-9141-3f772eb85e54) lcm/1* error idle 10.1.47.96 9999/TCP crash loop backoff: back-off 5m0s restarting failed container=lcm pod=lcm-559f79cfbb-blcsj_osm(1bc7397b-28b7-448a-b216-901f52336f78) mariadb/0* active idle 10.1.47.84 3306/TCP ready mon/2* active idle 10.1.47.116 8000/TCP ready mongodb/0* active idle 10.1.47.82 27017/TCP nbi/0* active idle 10.1.47.121 9999/TCP ready nbi/1 waiting idle 10.1.47.92 9999/TCP waiting for container ng-ui/1* active idle 10.1.47.90 80/TCP ready pla/1* active idle 10.1.47.125 9999/TCP ready pol/1* error idle 10.1.47.77 9999/TCP crash loop backoff: back-off 5m0s restarting failed container=pol pod=pol-799dbfc869-qjv9h_osm(8b048123-9354-4cda-b4c0-6875ee90aa48) prometheus/0* active idle 10.1.47.123 9090/TCP ready ro/1* active idle 10.1.47.78 9090/TCP ready zookeeper/0 error lost 10.1.47.95 container error: Could you help for this problem ? How to reproduce the problem: 1. Install OSM: wget https://osm-download.etsi.org/ftp/osm-13.0-thirteen/install_osm.sh chmod +x install_osm.sh ./install_osm.sh --charmed 2. Upgrade controller juju upgrade-controller --agent-version 2.9.37 3. Upgrade model juju upgrade-model --agent-version 2.9.37 4. Check the status of applications juju status, juju debug-log Many Thanks! Gulsum
2023-01-18 04:34:44 Ian Booth marked as duplicate 1997253