sysinv 2020-07-27 01:22:05.842 87629 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0] sysinv 2020-07-27 01:22:05.946 87629 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:1, cores/socket=6, threads/core=1, reference:current (unchanged) sysinv 2020-07-27 01:22:05.947 87629 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 sysinv 2020-07-27 01:22:05.947 87629 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 sysinv 2020-07-27 01:22:05.947 87629 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 sysinv 2020-07-27 01:22:05.948 87629 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 sysinv 2020-07-27 01:22:09.800 87629 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': -1, 'name': u'enp2s1'}] sysinv 2020-07-27 01:22:09.813 8361 INFO sysinv.agent.manager [-] Affine platform interface enp2s1 with cpulist 0,1 sysinv 2020-07-27 01:22:09.870 8361 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp2s1 0,1 return 0 sysinv 2020-07-27 01:22:09.932 8361 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:41a3dcad5feb sysinv 2020-07-27 01:22:09.942 87629 INFO sysinv.conductor.manager [-] Updating platform data for host: 90b6f5e3-7d11-4f73-bf97-856f2df26eaa with: {u'config_applied': u'738c199e-dde7-4d15-ae4c-5a2f84247e68', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:41a3dcad5feb'} sysinv 2020-07-27 01:22:10.054 87629 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 738c199e-dde7-4d15-ae4c-5a2f84247e68 sysinv 2020-07-27 01:22:10.105 8361 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': '738c199e-dde7-4d15-ae4c-5a2f84247e68', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:41a3dcad5feb'} sysinv 2020-07-27 01:22:15.185 8361 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0: 600 ... large disk defaults sysinv 2020-07-27 01:22:15.246 87629 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-07-27 01:22:15.248 87629 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-07-27 01:22:15.248 87629 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-07-27 01:22:15.248 87629 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-07-27 01:22:15.289 8361 INFO sysinv.agent.manager [-] Initial inventory completed host 90b6f5e3-7d11-4f73-bf97-856f2df26eaa sysinv 2020-07-27 01:22:47.068 87629 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '2244cd37-69de-4e6c-b414-759c611cfd31'. sysinv 2020-07-27 01:22:47.106 87629 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 2244cd37-69de-4e6c-b414-759c611cfd31, applied 738c199e-dde7-4d15-ae4c-5a2f84247e68 sysinv 2020-07-27 01:22:47.107 87629 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 738c199e-dde7-4d15-ae4c-5a2f84247e68 vs. target: 2244cd37-69de-4e6c-b414-759c611cfd31. sysinv 2020-07-27 01:22:47.144 87629 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=2244cd37-69de-4e6c-b414-759c611cfd31 sysinv 2020-07-27 01:22:47.145 87629 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=2244cd37-69de-4e6c-b414-759c611cfd31, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'] sysinv 2020-07-27 01:22:47.163 87629 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 2244cd37-69de-4e6c-b414-759c611cfd31 sysinv 2020-07-27 01:22:49.365 87629 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-07-27 01:22:49.366 87629 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-07-27 01:22:52.064 87629 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-07-27 01:22:52.065 87629 INFO sysinv.conductor.manager [-] No override change after configuration action, skipping re-apply of platform-integ-apps sysinv 2020-07-27 01:22:52.068 87629 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-07-27 01:22:52.069 87629 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 2244cd37-69de-4e6c-b414-759c611cfd31 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'90b6f5e3-7d11-4f73-bf97-856f2df26eaa']} to agent sysinv 2020-07-27 01:22:52.085 8361 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 2244cd37-69de-4e6c-b414-759c611cfd31 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'90b6f5e3-7d11-4f73-bf97-856f2df26eaa']} controller sysinv 2020-07-27 01:22:52.087 8361 INFO sysinv.agent.manager [-] controller-active sysinv 2020-07-27 01:22:52.087 8361 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-07-27 01:22:52.235 87629 INFO sysinv.conductor.manager [-] drbd-overview: pgsql-20.0, platform-9.8, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribution-16.0 sysinv 2020-07-27 01:22:52.237 87629 INFO sysinv.conductor.manager [-] lvdisplay: pgsql-20.0, platform-10.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribution-16.0 sysinv 2020-07-27 01:22:52.261 87629 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 738c199e-dde7-4d15-ae4c-5a2f84247e68 sysinv 2020-07-27 01:23:04.800 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-23-04 patch sysinv 2020-07-27 01:23:04.801 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:23:04.852 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-23-04 patch sysinv 2020-07-27 01:23:04.852 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-07-27 01:23:04.940 89073 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-07-27-01-23-04 patch sysinv 2020-07-27 01:23:05.751 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-23-05 patch sysinv 2020-07-27 01:23:05.751 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-07-27 01:23:05.752 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-enabled sysinv 2020-07-27 01:23:05.752 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-enabled ihost_val_prenotify: {} ihost_val: {} sysinv 2020-07-27 01:23:05.752 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-enabled, notify_vim=False notify_mtce=False rc=True sysinv 2020-07-27 01:23:05.753 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-07-27 01:23:05.753 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-enabled sysinv 2020-07-27 01:23:05.753 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 received services-enabled task= vim_progress_status=services-enabled sysinv 2020-07-27 01:23:05.753 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-enabled sysinv 2020-07-27 01:23:05.754 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-07-27 01:23:05.754 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-07-27 01:23:05.754 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-07-27 01:23:05.897 89073 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-07-27-01-23-05 patch sysinv 2020-07-27 01:24:03.047 89073 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-07-27 01:24:03.124 87629 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app stx-openstack sysinv 2020-07-27 01:24:03.402 87629 INFO sysinv.conductor.kube_app [-] Application stx-openstack (1.0-19-centos-stable-latest) apply started. sysinv 2020-07-27 01:24:04.264 87629 INFO sysinv.conductor.kube_app [-] Secret default-registry-key created under Namespace kube-system. sysinv 2020-07-27 01:24:04.367 87629 INFO sysinv.conductor.kube_app [-] Secret default-registry-key created under Namespace openstack. sysinv 2020-07-27 01:24:04.432 87629 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-07-27 01:24:06.411 87629 INFO sysinv.helm.neutron [-] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'physnet0:1500,physnet1:1500'}, 'ml2_type_flat': {'flat_networks': ''}} sysinv 2020-07-27 01:24:17.407 87629 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/stx-openstack/1.0-19-centos-stable-latest/stx-openstack-stx-openstack-del.yaml generated sysinv 2020-07-27 01:24:17.411 87629 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-07-27 01:24:18.966 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-07-27 01:24:18.983 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ceilometer:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:18.986 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-cinder:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:18.991 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-keystone-api-proxy:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:18.994 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/mariadb:10.2.13 download started from local registry sysinv 2020-07-27 01:24:28.757 8361 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime']. sysinv 2020-07-27 01:24:28.762 8361 INFO sysinv.agent.manager [-] Agent config applied 2244cd37-69de-4e6c-b414-759c611cfd31 sysinv 2020-07-27 01:24:28.865 87629 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 2244cd37-69de-4e6c-b414-759c611cfd31 sysinv 2020-07-27 01:24:29.557 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ceilometer:rc-3.0-centos-stable-latest download succeeded in 10 seconds sysinv 2020-07-27 01:24:29.559 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-gnocchi:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:30.232 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download succeeded in 11 seconds sysinv 2020-07-27 01:24:30.235 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 download started from local registry sysinv 2020-07-27 01:24:30.309 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-cinder:rc-3.0-centos-stable-latest download succeeded in 11 seconds sysinv 2020-07-27 01:24:30.311 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-libvirt:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:31.138 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 download succeeded in 0 seconds sysinv 2020-07-27 01:24:31.139 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-heat:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:31.289 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/mariadb:10.2.13 download succeeded in 12 seconds sysinv 2020-07-27 01:24:31.290 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7.13 download started from local registry sysinv 2020-07-27 01:24:31.300 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-keystone-api-proxy:rc-3.0-centos-stable-latest download succeeded in 12 seconds sysinv 2020-07-27 01:24:31.301 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-fm-rest-api:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:36.336 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-libvirt:rc-3.0-centos-stable-latest download succeeded in 6 seconds sysinv 2020-07-27 01:24:36.338 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-gnocchi:rc-3.0-centos-stable-latest download succeeded in 6 seconds sysinv 2020-07-27 01:24:36.339 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/openstackhelm/mariadb:10.2.18 download started from local registry sysinv 2020-07-27 01:24:36.343 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-barbican:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:41.682 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-heat:rc-3.0-centos-stable-latest download succeeded in 10 seconds sysinv 2020-07-27 01:24:41.685 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7.13 download succeeded in 10 seconds sysinv 2020-07-27 01:24:41.686 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/memcached:1.5.5 download started from local registry sysinv 2020-07-27 01:24:41.690 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-mariadb:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:47.340 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/openstackhelm/mariadb:10.2.18 download succeeded in 11 seconds sysinv 2020-07-27 01:24:47.343 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-keystone:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:47.349 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-fm-rest-api:rc-3.0-centos-stable-latest download succeeded in 16 seconds sysinv 2020-07-27 01:24:47.351 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/gcr.io/google_containers/defaultbackend:1.0 download started from local registry sysinv 2020-07-27 01:24:47.868 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-barbican:rc-3.0-centos-stable-latest download succeeded in 11 seconds sysinv 2020-07-27 01:24:47.869 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-nova-api-proxy:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:48.028 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/memcached:1.5.5 download succeeded in 6 seconds sysinv 2020-07-27 01:24:48.029 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:53.078 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/gcr.io/google_containers/defaultbackend:1.0 download succeeded in 5 seconds sysinv 2020-07-27 01:24:53.082 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-keystone:rc-3.0-centos-stable-latest download succeeded in 5 seconds sysinv 2020-07-27 01:24:53.083 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-nova:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:53.089 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-mariadb:rc-3.0-centos-stable-latest download succeeded in 11 seconds sysinv 2020-07-27 01:24:53.090 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-aodh:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:24:53.093 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 download started from local registry sysinv 2020-07-27 01:24:53.655 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-nova-api-proxy:rc-3.0-centos-stable-latest download succeeded in 5 seconds sysinv 2020-07-27 01:24:53.656 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/nginx:1.13.3 download started from local registry sysinv 2020-07-27 01:25:03.872 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest download succeeded in 15 seconds sysinv 2020-07-27 01:25:03.877 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-nova:rc-3.0-centos-stable-latest download succeeded in 10 seconds sysinv 2020-07-27 01:25:03.879 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 download started from local registry sysinv 2020-07-27 01:25:03.884 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-horizon:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:09.282 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 download succeeded in 16 seconds sysinv 2020-07-27 01:25:09.303 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ovs:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:09.339 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-aodh:rc-3.0-centos-stable-latest download succeeded in 16 seconds sysinv 2020-07-27 01:25:09.340 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7.13-management download started from local registry sysinv 2020-07-27 01:25:14.700 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/nginx:1.13.3 download succeeded in 21 seconds sysinv 2020-07-27 01:25:14.706 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 download succeeded in 10 seconds sysinv 2020-07-27 01:25:14.708 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-glance:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:14.714 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-panko:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:15.234 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-horizon:rc-3.0-centos-stable-latest download succeeded in 11 seconds sysinv 2020-07-27 01:25:15.235 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7-management download started from local registry sysinv 2020-07-27 01:25:15.533 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ovs:rc-3.0-centos-stable-latest download succeeded in 6 seconds sysinv 2020-07-27 01:25:15.534 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-placement:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:15.671 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7.13-management download succeeded in 6 seconds sysinv 2020-07-27 01:25:15.673 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ironic:rc-3.0-centos-stable-latest download started from local registry sysinv 2020-07-27 01:25:15.932 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-glance:rc-3.0-centos-stable-latest download succeeded in 1 seconds sysinv 2020-07-27 01:25:16.183 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/rabbitmq:3.7-management download succeeded in 0 seconds sysinv 2020-07-27 01:25:16.473 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-panko:rc-3.0-centos-stable-latest download succeeded in 1 seconds sysinv 2020-07-27 01:25:16.625 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-ironic:rc-3.0-centos-stable-latest download succeeded in 0 seconds sysinv 2020-07-27 01:25:16.880 87629 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/stx-placement:rc-3.0-centos-stable-latest download succeeded in 1 seconds sysinv 2020-07-27 01:25:16.882 87629 INFO sysinv.conductor.kube_app [-] All docker images for application stx-openstack were successfully downloaded in 57 seconds sysinv 2020-07-27 01:25:16.884 87629 INFO sysinv.conductor.kube_app [-] Wait if there are openstack charts in pending install... sysinv 2020-07-27 01:25:17.243 87629 INFO sysinv.conductor.kube_app [-] Starting Armada service... sysinv 2020-07-27 01:25:17.246 87629 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/19.12/admin.conf, manifests_dir=/opt/platform/armada/19.12, overrides_dir=/opt/platform/helm/19.12, logs_dir=/var/log/armada. sysinv 2020-07-27 01:25:17.888 87629 INFO sysinv.conductor.kube_app [-] Armada service started! sysinv 2020-07-27 01:25:17.890 87629 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable-chart-cleanup --debug /manifests/stx-openstack/1.0-19-centos-stable-latest/stx-openstack-stx-openstack.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/kube-system-ingress.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-ingress.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-nginx-ports-control.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-mariadb.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-garbd.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-memcached.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-rabbitmq.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-keystone.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-barbican.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-glance.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-cinder.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-ceph-rgw.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-libvirt.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-nova.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-nova-api-proxy.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-neutron.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-placement.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-heat.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-fm-rest-api.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-horizon.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-aodh.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-gnocchi.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-panko.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-ceilometer.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-dcdbsync.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-keystone-api-proxy.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-openvswitch.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/helm-toolkit-helm-toolkit.yaml --values /overrides/stx-openstack/1.0-19-centos-stable-latest/openstack-ironic.yaml --values /manifests/stx-openstack/1.0-19-centos-stable-latest/cg-openstack-ceph-rgw-meta.yaml --values /manifests/stx-openstack/1.0-19-centos-stable-latest/cg-openstack-mariadb-meta.yaml --values /manifests/stx-openstack/1.0-19-centos-stable-latest/cg-openstack-barbican-meta.yaml --values /manifests/stx-openstack/1.0-19-centos-stable-latest/cg-openstack-compute-kit-meta.yaml --values /manifests/stx-openstack/1.0-19-centos-stable-latest/cg-openstack-telemetry-meta.yaml --tiller-host tiller-deploy.kube-system.svc.cluster.local | tee /logs/stx-openstack-apply_2020-07-27-01-25-17.log' sysinv 2020-07-27 01:25:18.240 87629 INFO sysinv.conductor.kube_app [-] Starting progress monitoring thread for app stx-openstack sysinv 2020-07-27 01:25:23.205 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-kube-system-ingress, overall completion: 4.0% sysinv 2020-07-27 01:25:34.200 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-nginx-ports-control, overall completion: 11.0% sysinv 2020-07-27 01:25:58.435 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-mariadb, overall completion: 14.0% sysinv 2020-07-27 01:27:21.746 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-memcached, overall completion: 18.0% sysinv 2020-07-27 01:27:41.751 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-rabbitmq, overall completion: 21.0% sysinv 2020-07-27 01:28:37.408 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-keystone, overall completion: 25.0% sysinv 2020-07-27 01:30:29.502 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-glance, overall completion: 29.0% sysinv 2020-07-27 01:33:04.936 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-33-04 patch sysinv 2020-07-27 01:33:04.937 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:33:08.130 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-cinder, overall completion: 32.0% sysinv 2020-07-27 01:38:04.980 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-38-04 patch sysinv 2020-07-27 01:38:04.982 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:38:59.135 87629 INFO sysinv.conductor.kube_app [-] processing chart: osh-openstack-placement, overall completion: 54.0% sysinv 2020-07-27 01:43:05.014 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-43-04 patch sysinv 2020-07-27 01:43:05.015 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:48:05.023 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-48-04 patch sysinv 2020-07-27 01:48:05.024 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:53:05.043 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-53-05 patch sysinv 2020-07-27 01:53:05.044 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 01:58:05.354 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-01-58-05 patch sysinv 2020-07-27 01:58:05.355 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:03:05.141 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-03-05 patch sysinv 2020-07-27 02:03:05.142 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:08:05.376 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-08-05 patch sysinv 2020-07-27 02:08:05.377 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:09:14.316 87629 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-19-centos-stable-latest/stx-openstack-stx-openstack.yaml. See /var/log/armada/stx-openstack-apply_2020-07-27-01-25-17.log for details. sysinv 2020-07-27 02:09:14.320 87629 INFO sysinv.conductor.kube_app [-] Exiting progress monitoring thread for app stx-openstack sysinv 2020-07-27 02:09:14.517 87629 ERROR sysinv.conductor.kube_app [-] Application apply aborted!. sysinv 2020-07-27 02:09:14.518 87629 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app stx-openstack sysinv 2020-07-27 02:13:05.511 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-13-05 patch sysinv 2020-07-27 02:13:05.513 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:18:05.289 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-18-05 patch sysinv 2020-07-27 02:18:05.291 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:21:46.511 87629 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-07-27 02:21:46.554 87629 INFO sysinv.conductor.manager [-] Running tiller image: registry.local:9001/gcr.io/kubernetes-helm/tiller:v2.13.1 sysinv 2020-07-27 02:21:46.555 87629 INFO sysinv.conductor.manager [-] Requested tiller version: v2.13.1 sysinv 2020-07-27 02:21:46.556 87629 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-07-27 02:22:43.294 87629 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** changed: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** changed: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "msg": [] } TASK [common/push-docker-images : Log out of k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Log out of local registry] ******************* ok: [localhost] TASK [Create Calico config file] *********************************************** ok: [localhost] TASK [Update Calico Networking] ************************************************ changed: [localhost] TASK [Create Multus config file] *********************************************** ok: [localhost] TASK [Update Multus Networking] ************************************************ changed: [localhost] TASK [Create SRIOV Networking config file] ************************************* ok: [localhost] TASK [Update SRIOV Networking] ************************************************* changed: [localhost] TASK [Create SRIOV device plugin config file] ********************************** ok: [localhost] TASK [Update SRIOV device plugin] ********************************************** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=48 changed=15 unreachable=0 failed=0 . sysinv 2020-07-27 02:23:05.249 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-23-05 patch sysinv 2020-07-27 02:23:05.250 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:28:05.432 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-28-05 patch sysinv 2020-07-27 02:28:05.433 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-07-27 02:33:05.613 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-07-27-02-33-05 patch sysinv 2020-07-27 02:33:05.614 89073 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0.