Activity log for bug #1826912

Date Who What changed Old value New value Message
2019-04-29 17:54:46 Gopinath bug added bug
2019-04-29 17:54:46 Gopinath attachment added sysinv.log https://bugs.launchpad.net/bugs/1826912/+attachment/5260025/+files/sysinv.log
2019-04-29 17:58:28 Gopinath description Brief Description ----------------- When I try to apply Stx-openstack application using CMD(system host-apply stx-openstack), With Latest ISO and helm charts. The status hung at 70% and it fails. Severity -------- Provide the severity of the defect. <Critical: System/Feature is not usable after the defect>   Steps to Reproduce ------------------ Reference link: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard#Verify_the_cluster_endpoints 1. Configured Setup all nodes 2. Provisioning done for all nodes 3. Add ceph OSDs to controllers 4. Using sysinv to bring up/down the containerized services a. When doing Bring up services, used below command >> system application-apply stx-openstack >> system application-list Error: operation aborted, check logs for detail Expected Behavior ------------------ The application have to reach 100% and its should be in applied state.  Actual Behavior ---------------- The application hung at 70% and it in failed state. +---------------+-----------------+---------------+---------------+------------------------------------------+ | application | manifest name | manifest file | status | progress | +---------------+-----------------+---------------+---------------+------------------------------------------+ | stx-openstack | armada-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail | +---------------+-----------------+---------------+---------------+------------------------------------------+ Reproducibility --------------- <Reproducible/Intermittent> State if the issue is 100% reproducible.  System Configuration -------------------- <Multi-node system, Dedicated storage>   Branch/Pull Time/Commit ----------------------- bootimage.iso 2019-Apr-28 00:19:54 1.9G application/x-iso9660-image ISO: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190428T013000Z/outputs/iso/ HelmChart: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190428T013000Z/outputs/helm-charts/ helm-charts-manifest-centos-stable-latest.tgz Last Pass --------- Yes, it passed with older ISO http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190407T233001Z/outputs/ And use the helm chart from same folder Timestamp/Logs -------------- Attached logs of pods_status, sysinv,syslog and dmesg. Docker logs: [wrsroot@controller-0 ~(keystone_admin)]$ sudo docker exec armada_service tail -f stx-openstack-apply.log Password: 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2019-04-28 21:48:36.600 14 ERROR armada.cli raise self._exception 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2019-04-28 21:48:36.600 14 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 252, in handle 2019-04-28 21:48:36.600 14 ERROR armada.cli return armada.sync() 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 249, in sync 2019-04-28 21:48:36.600 14 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2019-04-28 21:48:36.600 14 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openvswitch', 'libvirt', 'neutron', 'nova'] 2019-04-28 21:48:36.600 14 ERROR armada.cli Brief Description ----------------- When I try to apply Stx-openstack application using CMD (system host-apply stx-openstack), With Latest ISO and helm charts. The status hung at 70% and it fails. Severity -------- Provide the severity of the defect. <Critical: System/Feature is not usable after the defect> Steps to Reproduce ------------------ Reference link:   https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard#Verify_the_cluster_endpoints   1. Configured Setup all nodes   2. Provisioning done for all nodes   3. Add ceph OSDs to controllers   4. Using sysinv to bring up/down the containerized services  a. When doing Bring up services, used below command   >> system application-apply stx-openstack   >> system application-list  Error: operation aborted, check logs for detail Expected Behavior ------------------ The application have to reach 100% and its should be in applied state. Actual Behavior ---------------- The application hung at 70% and it in failed state. +---------------+-----------------+---------------+---------------+------------------------------------------+ | application | manifest name | manifest file | status | progress | +---------------+-----------------+---------------+---------------+------------------------------------------+ | stx-openstack | armada-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail | +---------------+-----------------+---------------+---------------+------------------------------------------+ Reproducibility --------------- <Reproducible/Intermittent> State if the issue is 100% reproducible. System Configuration -------------------- <Multi-node system, Dedicated storage> Branch/Pull Time/Commit ----------------------- bootimage.iso 2019-Apr-28 00:19:54 1.9G application/x-iso9660-image ISO: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190428T013000Z/outputs/iso/ HelmChart: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190428T013000Z/outputs/helm-charts/ helm-charts-manifest-centos-stable-latest.tgz Last Pass --------- Yes, it passed with older ISO http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190407T233001Z/outputs/ And used the helm chart from same link above Timestamp/Logs -------------- Attached logs of pods_status, sysinv,syslog and dmesg. Docker logs: [wrsroot@controller-0 ~(keystone_admin)]$ sudo docker exec armada_service tail -f stx-openstack-apply.log Password: 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2019-04-28 21:48:36.600 14 ERROR armada.cli raise self._exception 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2019-04-28 21:48:36.600 14 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 252, in handle 2019-04-28 21:48:36.600 14 ERROR armada.cli return armada.sync() 2019-04-28 21:48:36.600 14 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 249, in sync 2019-04-28 21:48:36.600 14 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2019-04-28 21:48:36.600 14 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openvswitch', 'libvirt', 'neutron', 'nova'] 2019-04-28 21:48:36.600 14 ERROR armada.cli
2019-04-29 17:59:14 Gopinath attachment added dmesg https://bugs.launchpad.net/starlingx/+bug/1826912/+attachment/5260026/+files/dmesg
2019-04-29 18:21:39 Gopinath bug added subscriber Daniel Casal Quintero
2019-04-29 18:22:17 Gopinath bug added subscriber Elio Martinez
2019-05-01 18:28:53 Frank Miller starlingx: status New Triaged
2019-05-01 18:28:57 Frank Miller starlingx: importance Undecided High
2019-05-01 18:29:17 Frank Miller starlingx: assignee Cindy Xie (xxie1)
2019-05-01 18:31:37 Frank Miller tags stx.2.0 stx.containers
2019-05-07 05:06:10 Cindy Xie starlingx: assignee Cindy Xie (xxie1) yong hu (yhu6)
2019-05-07 12:17:04 yong hu attachment added 0428_build_multi-nodes.png https://bugs.launchpad.net/starlingx/+bug/1826912/+attachment/5262096/+files/0428_build_multi-nodes.png
2019-05-07 12:17:33 yong hu starlingx: status Triaged Incomplete
2019-05-08 01:55:07 yong hu attachment added stx-2+2+2-node-vm-specs.tgz https://bugs.launchpad.net/starlingx/+bug/1826912/+attachment/5262205/+files/stx-2+2+2-node-vm-specs.tgz
2019-05-21 14:46:04 chen haochuan attachment added sysinv.log https://bugs.launchpad.net/starlingx/+bug/1826912/+attachment/5265300/+files/sysinv.log
2019-05-30 02:19:26 yong hu starlingx: assignee yong hu (yhu6)
2019-05-30 02:20:34 yong hu starlingx: assignee Elio Martinez (elio1979)
2019-06-10 02:49:09 yong hu starlingx: assignee Elio Martinez (elio1979) yong hu (yhu6)
2019-06-10 02:49:15 yong hu starlingx: status Incomplete Fix Released